Re: [Twisted-Python] FTPClient storeFile help.
Hi, Glyph Lefkowitz wrote:
Oisin Mulvihill wrote:
I confess I really don't know how to handle the two deferred object that get returned by storeFile. I just add callback/ errorback for each, this could be the problem.
If you want to wait for both, use twisted.internet.defer.DeferredList.
Why are there two deferred objects returned by storeFile? The documentation says: storeFile (self ,path ): Store a file at the given path. This method issues the 'STOR' FTP command. Returns: A tuple of two Deferreds: * Deferred IFinishableConsumer . You must call the finish method on the IFinishableConsumer when the file is completely transferred. * Deferred list of control-connection responses. This really doesn't help me much. I get the impression that storeFile isn't sending the contents of file, because its waiting for me to stream the file contents somehow. Maybe via IFinishableConsumer? In my current program, using ftplib, I do the following to upload a file: : ftp.storbinary('STOR %s' % (filename), fp) : This has the filename and a file descriptor passed to it. Is something like this needed with storeFile? Please can someone give me an answer to this, as looking at the code for twisted.protocols.ftp isn't helping much either. This function seems to be a mystery to me and I want to understand how to use it properly. Thanks, om
Can anyone enlighten me as to what I'm doing wrong. I would really like to use proper async behavior for doing the uploads, instead of using threads + ftplib.
The FTP implementation in Twisted is sub-optimal (although that's mostly the server). If you don't get any satisfactory responses I recommend assigning a bug to spiv in the tracker :).
-- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540
On Fri, Oct 03, 2003 at 11:18:28AM +0100, Oisin Mulvihill wrote:
Hi,
[snip] Why are there two deferred objects returned by storeFile? The documentation says:
storeFile (self ,path ):
Store a file at the given path.
This method issues the 'STOR' FTP command.
Returns:
A tuple of two Deferreds:
* Deferred IFinishableConsumer . You must call the finish method on the IFinishableConsumer when the file is completely transferred.
* Deferred list of control-connection responses.
This really doesn't help me much. I get the impression that storeFile isn't sending the contents of file, because its waiting for me to stream the file contents somehow. Maybe via IFinishableConsumer?
Correct, I believe. It looks like you can use FileSender (from t.p.basic) if you're just sending something off disk (or from any file object, really) here.
In my current program, using ftplib, I do the following to upload a file:
: ftp.storbinary('STOR %s' % (filename), fp) :
This has the filename and a file descriptor passed to it. Is something like this needed with storeFile?
Please can someone give me an answer to this, as looking at the code for twisted.protocols.ftp isn't helping much either. This function seems to be a mystery to me and I want to understand how to use it properly.
Try... from twisted.protocols.basic import FileSender def sendfile(consumer, fileObj): s = FileSender() return s.beginFileTransfer(fileObj, consumer ).addCallback(lambda _: consumer.finish() ) dC, dL = p.storeFile("path") dC.addCallback(sendfile, fileObj) Jp
Jp Calderone wrote:
On Fri, Oct 03, 2003 at 11:18:28AM +0100, Oisin Mulvihill wrote:
Hi,
[snip] Why are there two deferred objects returned by storeFile? The documentation says:
storeFile (self ,path ):
Store a file at the given path.
This method issues the 'STOR' FTP command.
Returns:
A tuple of two Deferreds:
* Deferred IFinishableConsumer . You must call the finish method on the IFinishableConsumer when the file is completely transferred.
* Deferred list of control-connection responses.
This really doesn't help me much. I get the impression that storeFile isn't sending the contents of file, because its waiting for me to stream the file contents somehow. Maybe via IFinishableConsumer?
Correct, I believe. It looks like you can use FileSender (from t.p.basic) if you're just sending something off disk (or from any file object, really) here.
In my current program, using ftplib, I do the following to upload a file:
: ftp.storbinary('STOR %s' % (filename), fp) :
This has the filename and a file descriptor passed to it. Is something like this needed with storeFile?
Please can someone give me an answer to this, as looking at the code for twisted.protocols.ftp isn't helping much either. This function seems to be a mystery to me and I want to understand how to use it properly.
Try...
from twisted.protocols.basic import FileSender
def sendfile(consumer, fileObj): s = FileSender() return s.beginFileTransfer(fileObj, consumer ).addCallback(lambda _: consumer.finish() )
dC, dL = p.storeFile("path") dC.addCallback(sendfile, fileObj)
Jp
Thanks for replying. I've tried what you've suggested, however I'm still getting the code "waiting" when the transfer is supposed to happen. The follow is the code I added based on your suggestion (attached is the complete source): <snip> : : def sendfile(consumer, fileObj): """ """ print "2.1 consumer, fileObject -", consumer, fileObj s = FileSender() print "2.2" d = s.beginFileTransfer(fileObj, consumer) print "2.3" d.addCallback(lambda _: consumer.finish()) print "2.4" return d def connectionMade(ftpClient): """ """ try: print "1.0" my_file = StringIO("Hello! Did this get sent to the remote side?") print "1.1" my_filename = 'myfile.txt' print "1.2" ftpClient.pwd().addCallbacks(success, fail) print "1.3" dC, dL = ftpClient.storeFile(my_filename) print "1.4" dC.addCallback(sendfile, my_file) print "1.5" dL.addCallbacks(success, fail) print "1.6" except: print "Exception -", sys.exc_value reactor.stop() : : </snip> When I run this I get as far as the line: d = s.beginFileTransfer(fileObj, consumer) This is where it now seems to be hanginig. The file transfer doesn't appear to be happening. I noticed from the docs on beginFileTransfer(), that it takes 3 args: file object, consumer & transform function. I tried adding a transform function to see it gets called with the data, read from the file, however this never occured. I'm not really sure where to go from here. Any suggestions? Thanks, om -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540 # Twisted, the Framework of Your Internet # Copyright (C) 2001 Matthew W. Lefkowitz # # This library is free software; you can redistribute it and/or # modify it under the terms of version 2.1 of the GNU Lesser General Public # License as published by the Free Software Foundation. # # This library is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public # License along with this library; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA """ An example of using the FTP client """ # Twisted imports from twisted.protocols.ftp import FTPClient, FTPFileListProtocol from twisted.internet.protocol import Protocol, ClientCreator from twisted.protocols.basic import FileSender from twisted.python import usage from twisted.internet import reactor # Standard library imports import string import sys try: from cStringIO import StringIO except ImportError: from StringIO import StringIO # Define some callbacks def success(response): print 'Success! Got response:' print '---' if response is None: print None else: print string.join(response, '\n') print '---' def fail(error): print 'Failed. Error was:' print error from twisted.internet import reactor reactor.stop() class Options(usage.Options): optParameters = [['host', 'h', ''], ['port', 'p', 1221], ['username', 'u', 'anonymous'], ['password', None, 'anonumous@example.com'], ['passive', None, 0], ['debug', 'd', 1], ] def run(): # Get config config = Options() config.parseOptions() config.opts['port'] = int(config.opts['port']) config.opts['passive'] = int(config.opts['passive']) config.opts['debug'] = int(config.opts['debug']) # Create the client FTPClient.debug = config.opts['debug'] creator = ClientCreator(reactor, FTPClient, config.opts['username'], config.opts['password'], passive=config.opts['passive']) creator.connectTCP(config.opts['host'], config.opts['port']).addCallback(connectionMade) print "**** Here 1 -" reactor.run() print "- Here 2 ****" def sendfile(consumer, fileObj): """ """ print "2.1 consumer, fileObject -", consumer, fileObj s = FileSender() print "2.2" d = s.beginFileTransfer(fileObj, consumer) print "2.3" d.addCallback(lambda _: consumer.finish()) print "2.4" return d def connectionMade(ftpClient): """ """ try: print "1.0" my_file = StringIO("Hello! Did this get sent to the remote side?") print "1.1" my_filename = 'myfile.txt' print "1.2" ftpClient.pwd().addCallbacks(success, fail) print "1.3" dC, dL = ftpClient.storeFile(my_filename) print "1.4" dC.addCallback(sendfile, my_file) print "1.5" dL.addCallbacks(success, fail) print "1.6" except: print "Exception -", sys.exc_value reactor.stop() # this only runs if the module was *not* imported if __name__ == '__main__': run()
On Tue, Oct 07, 2003 at 01:20:06PM +0100, Oisin Mulvihill wrote:
Jp Calderone wrote:
[snip]
When I run this I get as far as the line:
d = s.beginFileTransfer(fileObj, consumer)
This is where it now seems to be hanginig. The file transfer doesn't appear to be happening. I noticed from the docs on beginFileTransfer(), that it takes 3 args: file object, consumer & transform function. I tried adding a transform function to see it gets called with the data, read from the file, however this never occured.
This suggests to me that the data connection is never actually being made. Try adding errbacks to all these deferreds and seeing if any of those are being called. Jp
Jp Calderone wrote:
On Tue, Oct 07, 2003 at 01:20:06PM +0100, Oisin Mulvihill wrote:
Jp Calderone wrote:
[snip]
When I run this I get as far as the line:
d = s.beginFileTransfer(fileObj, consumer)
This is where it now seems to be hanginig. The file transfer doesn't appear to be happening. I noticed from the docs on beginFileTransfer(), that it takes 3 args: file object, consumer & transform function. I tried adding a transform function to see it gets called with the data, read from the file, however this never occured.
This suggests to me that the data connection is never actually being made.
Try adding errbacks to all these deferreds and seeing if any of those are being called.
Jp
I'll look into it further, but as far as I can tell the connection is being made. An empty file does get created on the server side. I've I killed the ftp server and the errbacks get called, informing me of the connection loss. Would this indicate then that maybe the file sender isn't doing its job? I'll look into this as well. Any further suggestions would be welcome, Thanks, om -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540
Jp Calderone wrote:
On Tue, Oct 07, 2003 at 01:20:06PM +0100, Oisin Mulvihill wrote:
Jp Calderone wrote:
[snip]
When I run this I get as far as the line:
d = s.beginFileTransfer(fileObj, consumer)
This is where it now seems to be hanginig. The file transfer doesn't appear to be happening. I noticed from the docs on beginFileTransfer(), that it takes 3 args: file object, consumer & transform function. I tried adding a transform function to see it gets called with the data, read from the file, however this never occured.
This suggests to me that the data connection is never actually being made.
Try adding errbacks to all these deferreds and seeing if any of those are being called.
Jp
I forgot to mention I've tried this on my linux system with python 2.2.3 and twisted 1.0.7, with no success. om -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540
On Fri, Oct 03, 2003 at 11:18:28AM +0100, Oisin Mulvihill wrote:
Why are there two deferred objects returned by storeFile? The documentation says:
Try looking at twisted/test/test_ftp.py, line 251 (the testStor method of FTPClientAndServerTests). Hopefully that should help fill in the blanks. -Andrew.
Andrew Bennetts wrote:
On Fri, Oct 03, 2003 at 11:18:28AM +0100, Oisin Mulvihill wrote:
Why are there two deferred objects returned by storeFile? The documentation says:
Try looking at twisted/test/test_ftp.py, line 251 (the testStor method of FTPClientAndServerTests). Hopefully that should help fill in the blanks.
-Andrew.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hi, I know its been a while, but the project I needed this for has gone on the back burner. I was going to do an example for how I got the file to send and then do a recieve. I haven't got around to it. So for posterity and for anyone else who hits the same problem here is the modified ftp example that shows how I uploaded a file using twisted. The answer was in the unit tests I'd just forgotten twisted had them. All the best, om -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540 # Twisted, the Framework of Your Internet # Copyright (C) 2001 Matthew W. Lefkowitz # # This library is free software; you can redistribute it and/or # modify it under the terms of version 2.1 of the GNU Lesser General Public # License as published by the Free Software Foundation. # # This library is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public # License along with this library; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA """ An example of using the FTP client """ # Twisted imports from twisted.protocols.ftp import FTPClient, FTPFileListProtocol from twisted.internet.protocol import Protocol, ClientCreator from twisted.protocols.basic import FileSender from twisted.python import usage from twisted.internet import reactor # Standard library imports import string import sys try: from cStringIO import StringIO except ImportError: from StringIO import StringIO # Define some callbacks def success(response): print 'Success! Got response:' print '---' if response is None: print None else: print string.join(response, '\n') print '---' def fail(error): print 'Failed. Error was:' print error from twisted.internet import reactor reactor.stop() class Options(usage.Options): optParameters = [['host', 'h', ''], ['port', 'p', 1221], ['username', 'u', 'anonymous'], ['password', None, 'anonymous@example.com'], ['passive', None, 0], ['debug', 'd', 1], ] def run(): # Get config config = Options() config.parseOptions() config.opts['port'] = int(config.opts['port']) config.opts['passive'] = int(config.opts['passive']) config.opts['debug'] = int(config.opts['debug']) # Create the client FTPClient.debug = config.opts['debug'] creator = ClientCreator(reactor, FTPClient, config.opts['username'], config.opts['password'], passive=config.opts['passive']) creator.connectTCP(config.opts['host'], config.opts['port']).addCallback(connectionMade) print "**** FTP start -" reactor.run() print "- FTP stop ****" def sendfile(consumer, fileObj): """Its now time to send the file, so read and write it calling finish when we've finished. """ print "2.1 consumer, fileObject -", consumer, fileObj consumer.write(fileObj.read()) print "2.2" consumer.finish() print "2.3" reactor.stop() print "2.4 Exit Time." def connectionMade(ftpClient): """Ok, we're connected to the server. """ try: print "1.0" my_file = StringIO("Hello! Did this get sent to the remote side?") print "1.1" my_filename = 'myfile.txt' print "1.2" ftpClient.pwd().addCallbacks(success, fail) print "1.3" dC, dL = ftpClient.storeFile(my_filename) print "1.4" dC.addCallback(sendfile, my_file) print "1.5" dL.addCallbacks(success, fail) print "1.6" except: print "Exception -", sys.exc_value reactor.stop() # this only runs if the module was *not* imported if __name__ == '__main__': run()
participants (3)
-
Andrew Bennetts -
Jp Calderone -
Oisin Mulvihill