Mailman 3 October 2008 - Twisted

[Twisted-Python] troubles with FTP server and async file writing
by Brian Warner Oct. 7, 2008

Oct. 7, 2008

Howdy all.. who's interested in the VFS or FTP code these days? I'm in the process of writing an FTP frontend for Tahoe[1], and the STOR command is causing me problems. Tahoe is a virtual filesystem that spreads files and directories over a grid of backend servers to achieve high reliability and availability. Once the STOR command finishes transferring the file (over the separate DTP connection), the Tahoe node springs into action: encrypting, hashing, erasure-coding, uploading, and finally … [View More]modifying a parent directory (which is also distributed) to reference the new child. This whole process takes time, maybe a second or two for a small file, potentially minutes if you're uploading a large file over a slow network. Naturally, the Tahoe upload() method returns a Deferred, that fires when the whole process is complete. The trouble is, it looks like Twisted's FTP server is not designed to handle asynchronous uploads. The specific problem is in the way that twisted.protocols.ftp.FTP.ftp_STOR() interacts with the IConsumer that it gets.. the sequence goes like this: 1: ftp_STOR fetches an IWriteFile instance (via Deferred) 2: cbOpened() calls IWriteFile.receive to get an IConsumer (via Deferred) 3: cbConsumer() hands it to dtpInstance.registerConsumer (via Deferred) 4: FTP.registerConsumer() sends it a registerProducer() (synchronous) it returns the self._onConnLost Deferred 5: data is sent over the DTP connection, written to the consumer 6: the DTP connection is closed, _onConnLost is fired 7: DTP._unregConsumer sends unregisterProducer(), and ignores the result 8: FTP.cbConsumer()'s chain regains control, runs cbSent (via Deferred) 9: cbSent returns 226 Transfer Complete 10: the Deferred chain finally unwinds, firing the Deferred that was returned from ftp_STOR, which sends the 226 back to the client Most of these steps use Deferreds, so the opening of the file can take as long as we need, but since DTP._unregConsumer ignores the result of consumer.unregisterProducer in step #7, there's no way for the backend to stall the delivery of the 226 until the file has actually been uploaded. We need something to address this, because FTP clients are correctly assuming that once they get the 226 Transfer Complete, the file will be in place. I'm not sure how to best fix this. It seems like something needs to interact with either the IConsumer or the IWriteFile to ask "are you done uploading yet". If it wouldn't be so horribly wrong I'd suggest redefining IConsumer's unregisterProducer() to return a Deferred, and then modify the FTP client (in DTP._unregConsumer) to pass that Deferred back to the chain, something like: --- twisted/protocols/ftp.py (revision 24956) +++ twisted/protocols/ftp.py (working copy) @@ -451,10 +451,10 @@ self._buffer.append(bytes) def _unregConsumer(self, ignored): - self._cons.unregisterProducer() + d = defer.maybeDeferred(self._cons.unregisterProducer) self._cons = None del self._onConnLost - return ignored + return d But, as far as I can tell, IConsumers aren't meant to work this way: detaching a producer doesn't really mean that there won't be more data at some point in the future, just that this particular producer is no longer useful. IFinishableConsumer, though, has a 'finish' method, which is actually used in a couple of other places in twisted/protocols/ftp.py . So Plan B would be: * declare that IWriteFile.receive() must return an IFinishableConsumer instead of merely an IConsumer * document that IFinishableConsumer.finish() will be called after unregisterProducer, and document what happens in the case of exceptions * redefine IFinishableConsumer.finish() to be allowed to return a Deferred * modify FTP.ftp_STOR.cbConsumer to wait for this Deferred before sending the 226 response, something like: --- twisted/protocols/ftp.py (revision 24956) +++ twisted/protocols/ftp.py (working copy) @@ -1049,6 +1049,7 @@ cons = ASCIIConsumerWrapper(cons) d = self.dtpInstance.registerConsumer(cons) + d.addCallback(lambda res: cons.finish()) d.addCallbacks(cbSent, ebSent) # Tell them what to doooo But it doesn't look like IFinishableConsumer.finish() is meant to be used this way either: the docstring has no mention of return value, or if/when it gets called, and the FTP client is the only code in all of Twisted that uses it (and all that does is a synchronous transport.loseConnection). I guess Plan C would be to involve the IWriteFile instance, adding a finish() method of some sort, making this an FTP-specific solution. So, any thoughts? Something like this just at the VFS level would allow backends to use asynchronous-writes, which feels like a significant gap in the current functionality. It might be better to address it at the IConsumer level, but that would touch more (and better-established) code. cheers, -Brian [1]: http://allmydata.org/ [View Less]

2 2

Re: [Twisted-Python] Re:got stuck again
by shweta mani Oct. 5, 2008

Oct. 5, 2008

hi all again got stuck folks! this time it is as follows i need to execute a script called test1.sh which acepts a name from a user . this is done in a Unix server. sh test1.sh if executed will ask for a input. user will enter a name. then user's name will be displayed in the output form this should be done Python twisted ssh. in my case it is not accepting any input On Fri, Oct 3, 2008 at 9:13 PM, shweta mani <shweta3230(a)gmail.com> wrote: > hi all it really helped. > > … [View More]

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Oct. 5, 2008

Oct. 5, 2008

1 0

[Twisted-Python] CONNECTION_LOST not an integer (docstring error?)
by Michael McNeil Forbes Oct. 5, 2008

Oct. 5, 2008

Several places in the documentation refer to the method writeSomeData () returning a negative integer if the connection is lost. (In particular /api/ twisted.internet.abstract.FileDescriptor.html#writeSomeData.) However, twisted.internet.main.CONNECTION_LOST is often returned which is not an integer. The documentation and code should be brought into sync unless there is a more up-to-date reference that I am missing. Thanks, Michael.

3 8

[Twisted-Python] Implementing an efficient IMAP client using connection pooling
by little wiki Oct. 3, 2008

Oct. 3, 2008

Hi Folks, I'm implementing a web based interface to an IMAP server.My idea is simple as in my web client sending a request to the python script in cgi-bin running apache .The python script in turn fetches from the IMAP and sends back JSON data to the web client. Here I want my python script to use twisted's IMAP's implementation .In general ,everytime I send a request to the IMAP server, it sends the data of the entire folder back which becomes a memory /bandwidth hog.What I'm looking for is … [View More]

2 1

[Twisted-Python] Twisted and SNMP
by Vincent Bernat Oct. 3, 2008

Oct. 3, 2008

Hi folks! For people wanting to use Twisted to do some SNMP queries, there are actually several solutions: - TwistedSNMP that depends on PySNMP, a pure Python implementation - pynetsnmp, a ctypes based implementation I was using pynetsnmp but it segfaults on Python 2.4, even with a recent ctypes. I have then rewrite something similar in pure C. It works fine for me and people are free to use it: https://trac.luffy.cx/wiremaps/wiki/SnmpLibrary It is feature limited: GET, GETNEXT … [View More]

2 1

[Twisted-Python] Ampoule - 0.0.3
by Valentino Volonghi Oct. 1, 2008

Oct. 1, 2008

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Today I release a new minor release for ampoule, my Twisted Matrix process pool. What's new: Ampoule 0.0.3 (2008-10-01) ========================== Features - -------- - Added a Twisted Matrix plugin to run a process pool service that can be used over the network using an AMP client. - Added ability to change the reactor used to run the children Changes - ------- - If you wrote your own bootstrap code for the pool you should change … [View More]

1 1