Mailman 3 May 2006 - Twisted

[Twisted-Python] [patch] 1/4 process disconneting
by Andrea Arcangeli May 19, 2006

May 19, 2006

Here a list of fixes present in CPUShare-Twisted mercurial tree and missing in Twisted SVN trunk. Most of these have been posted as a ticket in trac without feedback (by now the links in the tickets are obsolete due to bugs in tailor that required a rebuild of the SVN->HG repository, but nobody asked about the dangling link anyway). So I rensend them here just in case somebody is interested. I will not answer to replies because I've no time, sorry. Each email contains one patch. This fixes … [View More]a reentrance problem in connectionLost if invoked by shutdown at the same time as the real disconnect. diff -r 493b5c24e0f3 twisted/internet/process.py --- a/twisted/internet/process.py Tue May 16 04:57:00 2006 +0000 +++ b/twisted/internet/process.py Wed May 17 15:59:28 2006 +0200 @@ -82,7 +82,24 @@ def detectLinuxBrokenPipeBehavior(): # Call at import time detectLinuxBrokenPipeBehavior() -class ProcessWriter(abstract.FileDescriptor): +class ProcessReaderWriter(abstract.FileDescriptor): + """(Internal) Helper class to avoid code duplication between + ProcessReader and ProcessWriter.""" + def connectionLost(self, reason): + """Close my end of the pipe, signal the Process (which signals the + ProcessProtocol). + See also abstract.FileDescriptor.connectionLost. + """ + # connectionLost can be called multiple times, for example + # both from the loseConnection timer, and from the + # shutdown event as well, but childConnectionLost + # needs to be invoked only once + disconnected = self.disconnected + abstract.FileDescriptor.connectionLost(self, reason) + if not disconnected: + self.proc.childConnectionLost(self.name, reason) + +class ProcessWriter(ProcessReaderWriter): """(Internal) Helper class to write into a Process's input pipe. I am a helper which describes a selectable asynchronous writer to a @@ -174,14 +191,8 @@ class ProcessWriter(abstract.FileDescrip else: self.stopReading() - def connectionLost(self, reason): - """See abstract.FileDescriptor.connectionLost. - """ - abstract.FileDescriptor.connectionLost(self, reason) - self.proc.childConnectionLost(self.name, reason) - - -class ProcessReader(abstract.FileDescriptor): + +class ProcessReader(ProcessReaderWriter): """ProcessReader I am a selectable representation of a process's output pipe, such as @@ -224,13 +235,6 @@ class ProcessReader(abstract.FileDescrip self.disconnecting = 1 self.stopReading() self.reactor.callLater(0, self.connectionLost, failure.Failure(CONNECTION_DONE)) - - def connectionLost(self, reason): - """Close my end of the pipe, signal the Process (which signals the - ProcessProtocol). - """ - abstract.FileDescriptor.connectionLost(self, reason) - self.proc.childConnectionLost(self.name, reason) class Process(styles.Ephemeral): [View Less]

12 25

Re: Hilarious dream-logic (was Re: [Twisted-Python] [patch] (etc)
by Terry Jones May 19, 2006

May 19, 2006

>>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> On Thu, 18 May 2006 21:25:07 +0200, Andrea Arcangeli <andrea(a)cpushare.com> wrote: >> Overall using twisted for my project was a mistake. glyph> And yet, and yet, and yet... Right. That's the part that I find pretty odd. Andrea says that Twisted was a mistake for him, and that it has one severe/major core design problem: >>Overall using twisted for my project was a mistake. One major … [View More]core design >>problem it has is that it can't pass POLLERR|POLLHUP|POLLNVAL to the poll >>syscall unless POLLIN/POLLOUT are set too. This means that if I throttle >>the I/O I can't notice anymore when a disconnect happens. But the severe >>core design issues it has But, apparently, Twisted isn't that bad after all. It still seems like it's worth using. In fact, it's SO good why not FORK THE CODE just so we can quickly apply a few patches and keep using it? On top of generally being a pest, not being willing or making/having the time to follow well-established project guidelines, etc., Andrea decides not just to continue to use Twisted with a few of his own patches applied (which would save plenty of time), while putting more energy into getting his patches integrated, but, instead, to publicly announce a fork, make all the effort to put it online, set up a mailing list, etc. That all seems quite deliberately anti-social and deliberately disruptive and destructive, rather than constructive. Meanwhile I read that "CPUShare-Twisted will try to avoid duplication" - why yes, that's an excellent (i.e. belated and hypocritical) aim for a _forked_ project. It seems that just a little more effort (and a whole lot less than forking, publicizing, putting it online, running servers, svn, etc) could have avoided a hell of a lot more duplication. On top of this nastiness, repeatedly posting URLs of chat room discussion between a bunch of developers is pretty silly, and, again, smacks of a deliberately destructive attitude to a project which you are nevertheless happy to run off with in a fork. It's not as though the Twisted developers (of which I am not one) are trying to hide their IRC conversation, is it? In this day and age it's nice to see some people playing around, even swearing (oh the shock, the horror, the scandal), and nice to see them even unashamedly include it in their distribution. The most fruitful part of this thread, and what I think really makes it worth otherwise-productive developers spending their time replying, is that it's nice to see the reaction to Andrea's postings. As someone considering using Twisted, I find it interesting and encouraging to read this thread - it gives me increased confidence in Twisted, the people behind it, and the project in general. So while the thread is a waste for some, there is real value for others. My impression is the following: Andrea, having not followed project guidelines and rules because he doesn't have the time, having not been treated like a coding God for whom others will naturally rush to do the mundane tasks in order to receive his patches, and being unwilling to go the extra few yards to get his patches (eventually) incorporated, is now using his formerly precious time to go out of his way to undermine the project. Terry [View Less]

2 1

[Twisted-Python] status of twisted flow
by Maciek Kaminski May 19, 2006

May 19, 2006

What is the status of Twisted flow project? Project page (http://twistedmatrix.com/projects/flow/) states that it is unmaintained . Is it dead or alive? Maciej Kaminski

1 0

[Twisted-Python] how to "close" a TLS connection?
by Manlio Perillo May 17, 2006

May 17, 2006

Hi. I'm doing some unit test with TLS but I got: twisted.trial.util.DirtyReactorError: THIS WILL BECOME AN ERROR SOON! reactor left in unclean state, the following Selectables were left over: <<class 'twisted.internet.tcp.TLSConnection'> to ('localhost', 5432) at f0e590> All is ok if I do not use TLS (I simply close the connection). What's the problem? Thanks Manlio Perillo

2 1

[Twisted-Python] How to disconnect a client, when you only have an instance of pb.PBClientFactory ?
by Luc Stepniewski May 17, 2006

May 17, 2006

I'm still trying to kill a hung PB client connection to a server. I noticed the timeout parameter in reactor.connectTCP() is not what I what, as it is only used once the connection is established (PB handshake). I'm trying to kill a connection that didn't manage to make the PB handshake (for example when instead of having a PB server, you have a server that replies to TCP but not to PB, like a netcat program). So my idea is to launch a callLater that, if the connection is still present … [View More]

2 1

[Twisted-Python] Question about PB and multicast
by Chaz. May 17, 2006

May 17, 2006

I am building an application that has tens of thousands of clients. There are times when I need to talk to quite a few of the clients using PB. When I have to talk to a single (or a few) clients via PB I do the normal thing - issue multiple calls. In the cases where I have to talk to quite a few clients I had thought of changing the object I am sending to include the names of the machines I am interested in talking to. I would then send it via multicast using PB. I know this would require some … [View More]

3 3

Re: [Twisted-Python] Send many large files with PB
by Brian Warner May 17, 2006

May 17, 2006

Justin Mazzola Paluska <jmp(a)MIT.EDU> writes: > - Should I send the files from SRC to DEST one-by-one? That's how I would do it. If you're talking about gigabyte-sized files, the protocol overhead will be pretty minimal compared to the data being transferred. You've got a couple of objects to keep track of for each file being sent, but on the other hand it will be a lot easier to keep track of how much progress you've made (and keep the user informed) that way. > - Or, is it … [View More]

2 1

[Twisted-Python] Send many large files with PB
by Justin Mazzola Paluska May 17, 2006

May 17, 2006

Hi, I'm using PB in a distributed application that has suddenly grown the requirement to copy directories of files between the servers. >From lurking on the mailing list archives, it seems that the best way to move large amounts of data between Twisted servers is to use a twisted.spread.util.Pager sub-class to pipe the data. Between that information and the "How to use twisted pb pager" [1] document, I'm probably good to go on how to transfer large amounts of data. However, I wanted to … [View More]step back and ask what's the best way to actually package the files that I'm going to send. To make things concrete, suppose I need to send data from SRC to DEST and that SRC has a PB RemoteReference to DEST. Also, most files will be huge (gigabytes) and nested in directories. - Should I send the files from SRC to DEST one-by-one? That is, make a new PB request for a new Pager reference for each file, stream the file using a twisted.spread.util.FilePager instance, then repeat with the next file, and so on. This has the advantage that I think I can do it fairly easily, but has the disadvantage of requiring many PB calls (with the associated bookkeeping in my application). - Or, is it better to use something like tarfile module to create a stream of bytes that I stream to the other side and decode? There's something appealing to using tarfile--it's like the oft-seen "tar -xf - | ssh user@host 'tar -cf -'" way of transferring files. Plus, the tarfile module takes care of making directories and the like for me. This method has the advantage of a single PB call, but the disadvantage that I can't quite figure out how to use tarfile with Twisted. The tarfile module requires an file-like object to stream to or stream from. I don't think the naive approach of just adding __write__ method to a Pager or __read__ method to a CallbackPageCollector will work without taking up all of the memory in my system or blocking in some way. - Finally, should I be doing something completely different? Normally, outside of my application, I'd just use rsync, scp, or some such. However, the users of this application don't know how to use these tools. I can't spawn these programs without getting into authentication issues between the machines. Doing this within Twisted seems like a good idea because the machines are already authenticated to each other through PB, but I could be wrong. I apologize if this is rambling. I've been thinking about this for a while and am now a bit bleary-eyed. --Justin [1] http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/457670 [View Less]

2 3

[Twisted-Python] sAsync: SQLAlchemy Made Asynchronous
by Ed Suominen May 17, 2006

May 17, 2006

sAsync: SQLAlchemy Made Asynchronous ---------------------------------------- http://foss.eepatents.com/sAsync/ "sAsync offers asynchronous access to the outstanding SQLAlchemy package by Michael Bayer. It also provides SQLAlchemy enhancements including persistent dictionaries, text indexing and searching, and an access broker for conveniently managing database access, table setup, and transactions. Everything can be run in an asynchronous fashion using the Twisted framework and its deferred … [View More]

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com May 14, 2006

May 14, 2006

1 0