Mailman 3 May 2006 - Twisted

[Twisted-Python] [patch] 1/4 process disconneting
by Andrea Arcangeli 19 May '06

19 May '06

Here a list of fixes present in CPUShare-Twisted mercurial tree and missing in Twisted SVN trunk. Most of these have been posted as a ticket in trac without feedback (by now the links in the tickets are obsolete due to bugs in tailor that required a rebuild of the SVN->HG repository, but nobody asked about the dangling link anyway). So I rensend them here just in case somebody is interested. I will not answer to replies because I've no time, sorry. Each email contains one patch. This fixes a reentrance problem in connectionLost if invoked by shutdown at the same time as the real disconnect. diff -r 493b5c24e0f3 twisted/internet/process.py --- a/twisted/internet/process.py Tue May 16 04:57:00 2006 +0000 +++ b/twisted/internet/process.py Wed May 17 15:59:28 2006 +0200 @@ -82,7 +82,24 @@ def detectLinuxBrokenPipeBehavior(): # Call at import time detectLinuxBrokenPipeBehavior() -class ProcessWriter(abstract.FileDescriptor): +class ProcessReaderWriter(abstract.FileDescriptor): + """(Internal) Helper class to avoid code duplication between + ProcessReader and ProcessWriter.""" + def connectionLost(self, reason): + """Close my end of the pipe, signal the Process (which signals the + ProcessProtocol). + See also abstract.FileDescriptor.connectionLost. + """ + # connectionLost can be called multiple times, for example + # both from the loseConnection timer, and from the + # shutdown event as well, but childConnectionLost + # needs to be invoked only once + disconnected = self.disconnected + abstract.FileDescriptor.connectionLost(self, reason) + if not disconnected: + self.proc.childConnectionLost(self.name, reason) + +class ProcessWriter(ProcessReaderWriter): """(Internal) Helper class to write into a Process's input pipe. I am a helper which describes a selectable asynchronous writer to a @@ -174,14 +191,8 @@ class ProcessWriter(abstract.FileDescrip else: self.stopReading() - def connectionLost(self, reason): - """See abstract.FileDescriptor.connectionLost. - """ - abstract.FileDescriptor.connectionLost(self, reason) - self.proc.childConnectionLost(self.name, reason) - - -class ProcessReader(abstract.FileDescriptor): + +class ProcessReader(ProcessReaderWriter): """ProcessReader I am a selectable representation of a process's output pipe, such as @@ -224,13 +235,6 @@ class ProcessReader(abstract.FileDescrip self.disconnecting = 1 self.stopReading() self.reactor.callLater(0, self.connectionLost, failure.Failure(CONNECTION_DONE)) - - def connectionLost(self, reason): - """Close my end of the pipe, signal the Process (which signals the - ProcessProtocol). - """ - abstract.FileDescriptor.connectionLost(self, reason) - self.proc.childConnectionLost(self.name, reason) class Process(styles.Ephemeral):

12 25

Re: Hilarious dream-logic (was Re: [Twisted-Python] [patch] (etc)
by Terry Jones 18 May '06

18 May '06

>>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> On Thu, 18 May 2006 21:25:07 +0200, Andrea Arcangeli <andrea(a)cpushare.com> wrote: >> Overall using twisted for my project was a mistake. glyph> And yet, and yet, and yet... Right. That's the part that I find pretty odd. Andrea says that Twisted was a mistake for him, and that it has one severe/major core design problem: >>Overall using twisted for my project was a mistake. One major core design >>problem it has is that it can't pass POLLERR|POLLHUP|POLLNVAL to the poll >>syscall unless POLLIN/POLLOUT are set too. This means that if I throttle >>the I/O I can't notice anymore when a disconnect happens. But the severe >>core design issues it has But, apparently, Twisted isn't that bad after all. It still seems like it's worth using. In fact, it's SO good why not FORK THE CODE just so we can quickly apply a few patches and keep using it? On top of generally being a pest, not being willing or making/having the time to follow well-established project guidelines, etc., Andrea decides not just to continue to use Twisted with a few of his own patches applied (which would save plenty of time), while putting more energy into getting his patches integrated, but, instead, to publicly announce a fork, make all the effort to put it online, set up a mailing list, etc. That all seems quite deliberately anti-social and deliberately disruptive and destructive, rather than constructive. Meanwhile I read that "CPUShare-Twisted will try to avoid duplication" - why yes, that's an excellent (i.e. belated and hypocritical) aim for a _forked_ project. It seems that just a little more effort (and a whole lot less than forking, publicizing, putting it online, running servers, svn, etc) could have avoided a hell of a lot more duplication. On top of this nastiness, repeatedly posting URLs of chat room discussion between a bunch of developers is pretty silly, and, again, smacks of a deliberately destructive attitude to a project which you are nevertheless happy to run off with in a fork. It's not as though the Twisted developers (of which I am not one) are trying to hide their IRC conversation, is it? In this day and age it's nice to see some people playing around, even swearing (oh the shock, the horror, the scandal), and nice to see them even unashamedly include it in their distribution. The most fruitful part of this thread, and what I think really makes it worth otherwise-productive developers spending their time replying, is that it's nice to see the reaction to Andrea's postings. As someone considering using Twisted, I find it interesting and encouraging to read this thread - it gives me increased confidence in Twisted, the people behind it, and the project in general. So while the thread is a waste for some, there is real value for others. My impression is the following: Andrea, having not followed project guidelines and rules because he doesn't have the time, having not been treated like a coding God for whom others will naturally rush to do the mundane tasks in order to receive his patches, and being unwilling to go the extra few yards to get his patches (eventually) incorporated, is now using his formerly precious time to go out of his way to undermine the project. Terry

2 1

[Twisted-Python] status of twisted flow
by Maciek Kaminski 18 May '06

18 May '06

What is the status of Twisted flow project? Project page (http://twistedmatrix.com/projects/flow/) states that it is unmaintained . Is it dead or alive? Maciej Kaminski

1 0

[Twisted-Python] how to "close" a TLS connection?
by Manlio Perillo 17 May '06

17 May '06

Hi. I'm doing some unit test with TLS but I got: twisted.trial.util.DirtyReactorError: THIS WILL BECOME AN ERROR SOON! reactor left in unclean state, the following Selectables were left over: <<class 'twisted.internet.tcp.TLSConnection'> to ('localhost', 5432) at f0e590> All is ok if I do not use TLS (I simply close the connection). What's the problem? Thanks Manlio Perillo

2 1

[Twisted-Python] How to disconnect a client, when you only have an instance of pb.PBClientFactory ?
by Luc Stepniewski 17 May '06

17 May '06

I'm still trying to kill a hung PB client connection to a server. I noticed the timeout parameter in reactor.connectTCP() is not what I what, as it is only used once the connection is established (PB handshake). I'm trying to kill a connection that didn't manage to make the PB handshake (for example when instead of having a PB server, you have a server that replies to TCP but not to PB, like a netcat program). So my idea is to launch a callLater that, if the connection is still present after 5 seconds (the callLater hasn't been cancelled), will close the client connection. The problem is that I just have an instance of pb.PBClientFactory, and I cannot find in the forest of methods which one I could use to cut that connection (if that's possible). I attached a small test file, if my explanation was really clear :-) Thanks for your help, Luc

2 1

[Twisted-Python] Question about PB and multicast
by Chaz. 17 May '06

17 May '06

I am building an application that has tens of thousands of clients. There are times when I need to talk to quite a few of the clients using PB. When I have to talk to a single (or a few) clients via PB I do the normal thing - issue multiple calls. In the cases where I have to talk to quite a few clients I had thought of changing the object I am sending to include the names of the machines I am interested in talking to. I would then send it via multicast using PB. I know this would require some changes on the PB side, but I am wondering if anyone sees a problem doing this? Peace, Chaz.

3 3

Re: [Twisted-Python] Send many large files with PB
by Brian Warner 17 May '06

17 May '06

Justin Mazzola Paluska <jmp(a)MIT.EDU> writes: > - Should I send the files from SRC to DEST one-by-one? That's how I would do it. If you're talking about gigabyte-sized files, the protocol overhead will be pretty minimal compared to the data being transferred. You've got a couple of objects to keep track of for each file being sent, but on the other hand it will be a lot easier to keep track of how much progress you've made (and keep the user informed) that way. > - Or, is it better to use something like tarfile module to create a > stream of bytes that I stream to the other side and decode? I would recommend this approach if you had a bunch of small files. You want to run that 'tar cf - WHAT' child against a ProcessProtocol that reacted to dataReceived(data) by doing a rref.callRemote("moreDataForYou", data). You'd probably want to accumulate data into chunks of maybe 4k or so to increase efficiency. At the far end, your remote_moreDataForYou() call would write that data into the untarring ProcessProtocol. Take a look at doc/core/howto/process.xhtml for details on ProcessProtocols and reactor.spawnProcess. > - Finally, should I be doing something completely different? > Normally, outside of my application, I'd just use rsync, scp, or > some such. I'd certainly investigate this method if the most of the files you are sending are already in place on the far end. The bandwidth savings are worth the extra setup hassle. Is there a way to get rsync to speak to stdout/stdin instead of using a TCP socket? If so, you could spawnProcess('rsync') and proxy it to the far end over PB as with 'tar' above. Or, you could have your PB-connection-wielding process listen on a local TCP socket, then tell rsync to talk directly to that port, then do a socket-level proxy over PB to the far system. Also remember that scp (or rsync-over-ssh or tar|ssh, etc) will be doing better authentication than PB, since PB is all in cleartext. Many applications don't require confidentiality, but before you switch from ssh to straight PB you should be aware of what exactly you're giving up. <shameless plug> But, if you use NewPB, you get the strong authentication and confidentiality of ssh with all of the juicy RemoteReference model you've come to know and love from PB, check out NewPB[1] today. </shameless plug>. cheers, -Brian [1]: http://twistedmatrix.com/trac/wiki/NewPB

2 1

[Twisted-Python] Send many large files with PB
by Justin Mazzola Paluska 17 May '06

17 May '06

Hi, I'm using PB in a distributed application that has suddenly grown the requirement to copy directories of files between the servers. >From lurking on the mailing list archives, it seems that the best way to move large amounts of data between Twisted servers is to use a twisted.spread.util.Pager sub-class to pipe the data. Between that information and the "How to use twisted pb pager" [1] document, I'm probably good to go on how to transfer large amounts of data. However, I wanted to step back and ask what's the best way to actually package the files that I'm going to send. To make things concrete, suppose I need to send data from SRC to DEST and that SRC has a PB RemoteReference to DEST. Also, most files will be huge (gigabytes) and nested in directories. - Should I send the files from SRC to DEST one-by-one? That is, make a new PB request for a new Pager reference for each file, stream the file using a twisted.spread.util.FilePager instance, then repeat with the next file, and so on. This has the advantage that I think I can do it fairly easily, but has the disadvantage of requiring many PB calls (with the associated bookkeeping in my application). - Or, is it better to use something like tarfile module to create a stream of bytes that I stream to the other side and decode? There's something appealing to using tarfile--it's like the oft-seen "tar -xf - | ssh user@host 'tar -cf -'" way of transferring files. Plus, the tarfile module takes care of making directories and the like for me. This method has the advantage of a single PB call, but the disadvantage that I can't quite figure out how to use tarfile with Twisted. The tarfile module requires an file-like object to stream to or stream from. I don't think the naive approach of just adding __write__ method to a Pager or __read__ method to a CallbackPageCollector will work without taking up all of the memory in my system or blocking in some way. - Finally, should I be doing something completely different? Normally, outside of my application, I'd just use rsync, scp, or some such. However, the users of this application don't know how to use these tools. I can't spawn these programs without getting into authentication issues between the machines. Doing this within Twisted seems like a good idea because the machines are already authenticated to each other through PB, but I could be wrong. I apologize if this is rambling. I've been thinking about this for a while and am now a bit bleary-eyed. --Justin [1] http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/457670

2 3

[Twisted-Python] sAsync: SQLAlchemy Made Asynchronous
by Ed Suominen 17 May '06

17 May '06

sAsync: SQLAlchemy Made Asynchronous ---------------------------------------- http://foss.eepatents.com/sAsync/ "sAsync offers asynchronous access to the outstanding SQLAlchemy package by Michael Bayer. It also provides SQLAlchemy enhancements including persistent dictionaries, text indexing and searching, and an access broker for conveniently managing database access, table setup, and transactions. Everything can be run in an asynchronous fashion using the Twisted framework and its deferred processing capabilities." As the site goes on to say, this package is very much a work in progress, and it's not even close to a release yet. It has only the beginning of a test suite, none of the planned searching capability is close to working, and it needs some more packaging work, for starters. However, you can browse the package source files from the current SVN trunk on the Trac site, or check out the current SVN trunk as follows: svn co svn://eepatents.com/var/svn/sasync/trunk sasync I very much hope that SQLAlchemy and Twisted users (especially those interested in the integrated use of both projects) will offer submissions. That can be done by creating tickets with uploaded files, editing the Wiki page http://foss.eepatents.com/sAsync/wiki/Submissions, or (if you're serious about this), by requesting and obtaining SVN commit privileges for the project from me. It would be great to see some collaborate work getting done on this. I'm also interested in whether this would be a prospect for an add-on to SA itself rather than a separately maintained package. Best regards, Ed Suominen

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 14 May '06

14 May '06

Bug summary ______________________ Summary for 2006-05-07 through 2006-05-14 Bugs opened: 10 Bugs closed: 6 Total open bugs: 612 (+4) |== Type Changes |== Priority Changes |== Component Changes |Defect: +3 |Highest: -1 |Core: +0 |Enhancement: +1 |High: -1 |Names: +1 |Normal: +6 |Pb: -1 |Trial: +1 |Web2: +1 |Website: +1 |Words: +1 New / Reopened Bugs ______________________ ===== Highest ===== [#1709] defgen incorrectly propagates handled Failures to the outer Deferred (opened by exarkun) (CLOSED, fixed) defect core http://twistedmatrix.com/trac/ticket/1709 [#1710] Failures can be created in a broken state (opened by exarkun) (CLOSED, fixed) defect core http://twistedmatrix.com/trac/ticket/1710 ===== High ===== [#1711] PyLucene doesn't close write indexes properly (opened by exarkun) (CLOSED, invalid) defect core http://twistedmatrix.com/trac/ticket/1711 ===== Normal ===== [#1706] Make todo a decorator (opened by jml) defect trial http://twistedmatrix.com/trac/ticket/1706 [#1707] cannot edit wiki pages (opened by warner) defect website http://twistedmatrix.com/trac/ticket/1707 [#1708] twisted.names shouldn't stop listening when crappy data is sent (opened by radix) enhancement names http://twistedmatrix.com/trac/ticket/1708 [#1712] Twisted overview graph shows Xish (opened by ralphm) defect core http://twistedmatrix.com/trac/ticket/1712 [#1713] Web2 test failure on win32 (SSLServerTest.testLingeringClose) (opened by teratorn) defect web2 http://twistedmatrix.com/trac/ticket/1713 [#1714] Update Words' NEWS for 2.3 release (opened by ralphm) enhancement words http://twistedmatrix.com/trac/ticket/1714 [#1715] move vertex.juice into twisted.internet (opened by glyph) enhancement core http://twistedmatrix.com/trac/ticket/1715 Closed Bugs ______________________ ===== Highest ===== [#1710] Failures can be created in a broken state (opened by exarkun, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/1710 [#1709] defgen incorrectly propagates handled Failures to the outer Deferred (opened by exarkun, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/1709 [#1668] Update release automation to work with recent version changes (opened by radix, closed by radix, fixed) enhancement core http://twistedmatrix.com/trac/ticket/1668 ===== High ===== [#1022] Explain Deferred for users familiar with callbacks (opened by hypatia, closed by edsuom, fixed) defect core http://twistedmatrix.com/trac/ticket/1022 [#1711] PyLucene doesn't close write indexes properly (opened by exarkun, closed by exarkun, invalid) defect core http://twistedmatrix.com/trac/ticket/1711 ===== Normal ===== [#1684] release newpb 0.0.1 ! (opened by warner, closed by radix, fixed) enhancement pb http://twistedmatrix.com/trac/ticket/1684

1 0