Re: [Twisted-Python] FTPClient storeFile help.
![](https://secure.gravatar.com/avatar/4a58f201f6865057fdac810ad1481bcc.jpg?s=120&d=mm&r=g)
Hi, Glyph Lefkowitz wrote:
Why are there two deferred objects returned by storeFile? The documentation says: storeFile (self ,path ): Store a file at the given path. This method issues the 'STOR' FTP command. Returns: A tuple of two Deferreds: * Deferred IFinishableConsumer . You must call the finish method on the IFinishableConsumer when the file is completely transferred. * Deferred list of control-connection responses. This really doesn't help me much. I get the impression that storeFile isn't sending the contents of file, because its waiting for me to stream the file contents somehow. Maybe via IFinishableConsumer? In my current program, using ftplib, I do the following to upload a file: : ftp.storbinary('STOR %s' % (filename), fp) : This has the filename and a file descriptor passed to it. Is something like this needed with storeFile? Please can someone give me an answer to this, as looking at the code for twisted.protocols.ftp isn't helping much either. This function seems to be a mystery to me and I want to understand how to use it properly. Thanks, om
-- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540
![](https://secure.gravatar.com/avatar/56e4cc78ea7fcf3bb37888ebf23bc1f0.jpg?s=120&d=mm&r=g)
On Fri, Oct 03, 2003 at 11:18:28AM +0100, Oisin Mulvihill wrote:
Correct, I believe. It looks like you can use FileSender (from t.p.basic) if you're just sending something off disk (or from any file object, really) here.
Try... from twisted.protocols.basic import FileSender def sendfile(consumer, fileObj): s = FileSender() return s.beginFileTransfer(fileObj, consumer ).addCallback(lambda _: consumer.finish() ) dC, dL = p.storeFile("path") dC.addCallback(sendfile, fileObj) Jp
![](https://secure.gravatar.com/avatar/4a58f201f6865057fdac810ad1481bcc.jpg?s=120&d=mm&r=g)
Jp Calderone wrote:
Thanks for replying. I've tried what you've suggested, however I'm still getting the code "waiting" when the transfer is supposed to happen. The follow is the code I added based on your suggestion (attached is the complete source): <snip> : : def sendfile(consumer, fileObj): """ """ print "2.1 consumer, fileObject -", consumer, fileObj s = FileSender() print "2.2" d = s.beginFileTransfer(fileObj, consumer) print "2.3" d.addCallback(lambda _: consumer.finish()) print "2.4" return d def connectionMade(ftpClient): """ """ try: print "1.0" my_file = StringIO("Hello! Did this get sent to the remote side?") print "1.1" my_filename = 'myfile.txt' print "1.2" ftpClient.pwd().addCallbacks(success, fail) print "1.3" dC, dL = ftpClient.storeFile(my_filename) print "1.4" dC.addCallback(sendfile, my_file) print "1.5" dL.addCallbacks(success, fail) print "1.6" except: print "Exception -", sys.exc_value reactor.stop() : : </snip> When I run this I get as far as the line: d = s.beginFileTransfer(fileObj, consumer) This is where it now seems to be hanginig. The file transfer doesn't appear to be happening. I noticed from the docs on beginFileTransfer(), that it takes 3 args: file object, consumer & transform function. I tried adding a transform function to see it gets called with the data, read from the file, however this never occured. I'm not really sure where to go from here. Any suggestions? Thanks, om -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540 # Twisted, the Framework of Your Internet # Copyright (C) 2001 Matthew W. Lefkowitz # # This library is free software; you can redistribute it and/or # modify it under the terms of version 2.1 of the GNU Lesser General Public # License as published by the Free Software Foundation. # # This library is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public # License along with this library; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA """ An example of using the FTP client """ # Twisted imports from twisted.protocols.ftp import FTPClient, FTPFileListProtocol from twisted.internet.protocol import Protocol, ClientCreator from twisted.protocols.basic import FileSender from twisted.python import usage from twisted.internet import reactor # Standard library imports import string import sys try: from cStringIO import StringIO except ImportError: from StringIO import StringIO # Define some callbacks def success(response): print 'Success! Got response:' print '---' if response is None: print None else: print string.join(response, '\n') print '---' def fail(error): print 'Failed. Error was:' print error from twisted.internet import reactor reactor.stop() class Options(usage.Options): optParameters = [['host', 'h', ''], ['port', 'p', 1221], ['username', 'u', 'anonymous'], ['password', None, 'anonumous@example.com'], ['passive', None, 0], ['debug', 'd', 1], ] def run(): # Get config config = Options() config.parseOptions() config.opts['port'] = int(config.opts['port']) config.opts['passive'] = int(config.opts['passive']) config.opts['debug'] = int(config.opts['debug']) # Create the client FTPClient.debug = config.opts['debug'] creator = ClientCreator(reactor, FTPClient, config.opts['username'], config.opts['password'], passive=config.opts['passive']) creator.connectTCP(config.opts['host'], config.opts['port']).addCallback(connectionMade) print "**** Here 1 -" reactor.run() print "- Here 2 ****" def sendfile(consumer, fileObj): """ """ print "2.1 consumer, fileObject -", consumer, fileObj s = FileSender() print "2.2" d = s.beginFileTransfer(fileObj, consumer) print "2.3" d.addCallback(lambda _: consumer.finish()) print "2.4" return d def connectionMade(ftpClient): """ """ try: print "1.0" my_file = StringIO("Hello! Did this get sent to the remote side?") print "1.1" my_filename = 'myfile.txt' print "1.2" ftpClient.pwd().addCallbacks(success, fail) print "1.3" dC, dL = ftpClient.storeFile(my_filename) print "1.4" dC.addCallback(sendfile, my_file) print "1.5" dL.addCallbacks(success, fail) print "1.6" except: print "Exception -", sys.exc_value reactor.stop() # this only runs if the module was *not* imported if __name__ == '__main__': run()
![](https://secure.gravatar.com/avatar/56e4cc78ea7fcf3bb37888ebf23bc1f0.jpg?s=120&d=mm&r=g)
On Tue, Oct 07, 2003 at 01:20:06PM +0100, Oisin Mulvihill wrote:
This suggests to me that the data connection is never actually being made. Try adding errbacks to all these deferreds and seeing if any of those are being called. Jp
![](https://secure.gravatar.com/avatar/4a58f201f6865057fdac810ad1481bcc.jpg?s=120&d=mm&r=g)
Jp Calderone wrote:
I'll look into it further, but as far as I can tell the connection is being made. An empty file does get created on the server side. I've I killed the ftp server and the errbacks get called, informing me of the connection loss. Would this indicate then that maybe the file sender isn't doing its job? I'll look into this as well. Any further suggestions would be welcome, Thanks, om -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540
![](https://secure.gravatar.com/avatar/4a58f201f6865057fdac810ad1481bcc.jpg?s=120&d=mm&r=g)
Jp Calderone wrote:
I forgot to mention I've tried this on my linux system with python 2.2.3 and twisted 1.0.7, with no success. om -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Fri, Oct 03, 2003 at 11:18:28AM +0100, Oisin Mulvihill wrote:
Why are there two deferred objects returned by storeFile? The documentation says:
Try looking at twisted/test/test_ftp.py, line 251 (the testStor method of FTPClientAndServerTests). Hopefully that should help fill in the blanks. -Andrew.
![](https://secure.gravatar.com/avatar/4a58f201f6865057fdac810ad1481bcc.jpg?s=120&d=mm&r=g)
Andrew Bennetts wrote:
Hi, I know its been a while, but the project I needed this for has gone on the back burner. I was going to do an example for how I got the file to send and then do a recieve. I haven't got around to it. So for posterity and for anyone else who hits the same problem here is the modified ftp example that shows how I uploaded a file using twisted. The answer was in the unit tests I'd just forgotten twisted had them. All the best, om -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540 # Twisted, the Framework of Your Internet # Copyright (C) 2001 Matthew W. Lefkowitz # # This library is free software; you can redistribute it and/or # modify it under the terms of version 2.1 of the GNU Lesser General Public # License as published by the Free Software Foundation. # # This library is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public # License along with this library; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA """ An example of using the FTP client """ # Twisted imports from twisted.protocols.ftp import FTPClient, FTPFileListProtocol from twisted.internet.protocol import Protocol, ClientCreator from twisted.protocols.basic import FileSender from twisted.python import usage from twisted.internet import reactor # Standard library imports import string import sys try: from cStringIO import StringIO except ImportError: from StringIO import StringIO # Define some callbacks def success(response): print 'Success! Got response:' print '---' if response is None: print None else: print string.join(response, '\n') print '---' def fail(error): print 'Failed. Error was:' print error from twisted.internet import reactor reactor.stop() class Options(usage.Options): optParameters = [['host', 'h', ''], ['port', 'p', 1221], ['username', 'u', 'anonymous'], ['password', None, 'anonymous@example.com'], ['passive', None, 0], ['debug', 'd', 1], ] def run(): # Get config config = Options() config.parseOptions() config.opts['port'] = int(config.opts['port']) config.opts['passive'] = int(config.opts['passive']) config.opts['debug'] = int(config.opts['debug']) # Create the client FTPClient.debug = config.opts['debug'] creator = ClientCreator(reactor, FTPClient, config.opts['username'], config.opts['password'], passive=config.opts['passive']) creator.connectTCP(config.opts['host'], config.opts['port']).addCallback(connectionMade) print "**** FTP start -" reactor.run() print "- FTP stop ****" def sendfile(consumer, fileObj): """Its now time to send the file, so read and write it calling finish when we've finished. """ print "2.1 consumer, fileObject -", consumer, fileObj consumer.write(fileObj.read()) print "2.2" consumer.finish() print "2.3" reactor.stop() print "2.4 Exit Time." def connectionMade(ftpClient): """Ok, we're connected to the server. """ try: print "1.0" my_file = StringIO("Hello! Did this get sent to the remote side?") print "1.1" my_filename = 'myfile.txt' print "1.2" ftpClient.pwd().addCallbacks(success, fail) print "1.3" dC, dL = ftpClient.storeFile(my_filename) print "1.4" dC.addCallback(sendfile, my_file) print "1.5" dL.addCallbacks(success, fail) print "1.6" except: print "Exception -", sys.exc_value reactor.stop() # this only runs if the module was *not* imported if __name__ == '__main__': run()
![](https://secure.gravatar.com/avatar/56e4cc78ea7fcf3bb37888ebf23bc1f0.jpg?s=120&d=mm&r=g)
On Fri, Oct 03, 2003 at 11:18:28AM +0100, Oisin Mulvihill wrote:
Correct, I believe. It looks like you can use FileSender (from t.p.basic) if you're just sending something off disk (or from any file object, really) here.
Try... from twisted.protocols.basic import FileSender def sendfile(consumer, fileObj): s = FileSender() return s.beginFileTransfer(fileObj, consumer ).addCallback(lambda _: consumer.finish() ) dC, dL = p.storeFile("path") dC.addCallback(sendfile, fileObj) Jp
![](https://secure.gravatar.com/avatar/4a58f201f6865057fdac810ad1481bcc.jpg?s=120&d=mm&r=g)
Jp Calderone wrote:
Thanks for replying. I've tried what you've suggested, however I'm still getting the code "waiting" when the transfer is supposed to happen. The follow is the code I added based on your suggestion (attached is the complete source): <snip> : : def sendfile(consumer, fileObj): """ """ print "2.1 consumer, fileObject -", consumer, fileObj s = FileSender() print "2.2" d = s.beginFileTransfer(fileObj, consumer) print "2.3" d.addCallback(lambda _: consumer.finish()) print "2.4" return d def connectionMade(ftpClient): """ """ try: print "1.0" my_file = StringIO("Hello! Did this get sent to the remote side?") print "1.1" my_filename = 'myfile.txt' print "1.2" ftpClient.pwd().addCallbacks(success, fail) print "1.3" dC, dL = ftpClient.storeFile(my_filename) print "1.4" dC.addCallback(sendfile, my_file) print "1.5" dL.addCallbacks(success, fail) print "1.6" except: print "Exception -", sys.exc_value reactor.stop() : : </snip> When I run this I get as far as the line: d = s.beginFileTransfer(fileObj, consumer) This is where it now seems to be hanginig. The file transfer doesn't appear to be happening. I noticed from the docs on beginFileTransfer(), that it takes 3 args: file object, consumer & transform function. I tried adding a transform function to see it gets called with the data, read from the file, however this never occured. I'm not really sure where to go from here. Any suggestions? Thanks, om -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540 # Twisted, the Framework of Your Internet # Copyright (C) 2001 Matthew W. Lefkowitz # # This library is free software; you can redistribute it and/or # modify it under the terms of version 2.1 of the GNU Lesser General Public # License as published by the Free Software Foundation. # # This library is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public # License along with this library; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA """ An example of using the FTP client """ # Twisted imports from twisted.protocols.ftp import FTPClient, FTPFileListProtocol from twisted.internet.protocol import Protocol, ClientCreator from twisted.protocols.basic import FileSender from twisted.python import usage from twisted.internet import reactor # Standard library imports import string import sys try: from cStringIO import StringIO except ImportError: from StringIO import StringIO # Define some callbacks def success(response): print 'Success! Got response:' print '---' if response is None: print None else: print string.join(response, '\n') print '---' def fail(error): print 'Failed. Error was:' print error from twisted.internet import reactor reactor.stop() class Options(usage.Options): optParameters = [['host', 'h', ''], ['port', 'p', 1221], ['username', 'u', 'anonymous'], ['password', None, 'anonumous@example.com'], ['passive', None, 0], ['debug', 'd', 1], ] def run(): # Get config config = Options() config.parseOptions() config.opts['port'] = int(config.opts['port']) config.opts['passive'] = int(config.opts['passive']) config.opts['debug'] = int(config.opts['debug']) # Create the client FTPClient.debug = config.opts['debug'] creator = ClientCreator(reactor, FTPClient, config.opts['username'], config.opts['password'], passive=config.opts['passive']) creator.connectTCP(config.opts['host'], config.opts['port']).addCallback(connectionMade) print "**** Here 1 -" reactor.run() print "- Here 2 ****" def sendfile(consumer, fileObj): """ """ print "2.1 consumer, fileObject -", consumer, fileObj s = FileSender() print "2.2" d = s.beginFileTransfer(fileObj, consumer) print "2.3" d.addCallback(lambda _: consumer.finish()) print "2.4" return d def connectionMade(ftpClient): """ """ try: print "1.0" my_file = StringIO("Hello! Did this get sent to the remote side?") print "1.1" my_filename = 'myfile.txt' print "1.2" ftpClient.pwd().addCallbacks(success, fail) print "1.3" dC, dL = ftpClient.storeFile(my_filename) print "1.4" dC.addCallback(sendfile, my_file) print "1.5" dL.addCallbacks(success, fail) print "1.6" except: print "Exception -", sys.exc_value reactor.stop() # this only runs if the module was *not* imported if __name__ == '__main__': run()
![](https://secure.gravatar.com/avatar/56e4cc78ea7fcf3bb37888ebf23bc1f0.jpg?s=120&d=mm&r=g)
On Tue, Oct 07, 2003 at 01:20:06PM +0100, Oisin Mulvihill wrote:
This suggests to me that the data connection is never actually being made. Try adding errbacks to all these deferreds and seeing if any of those are being called. Jp
![](https://secure.gravatar.com/avatar/4a58f201f6865057fdac810ad1481bcc.jpg?s=120&d=mm&r=g)
Jp Calderone wrote:
I'll look into it further, but as far as I can tell the connection is being made. An empty file does get created on the server side. I've I killed the ftp server and the errbacks get called, informing me of the connection loss. Would this indicate then that maybe the file sender isn't doing its job? I'll look into this as well. Any further suggestions would be welcome, Thanks, om -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540
![](https://secure.gravatar.com/avatar/4a58f201f6865057fdac810ad1481bcc.jpg?s=120&d=mm&r=g)
Jp Calderone wrote:
I forgot to mention I've tried this on my linux system with python 2.2.3 and twisted 1.0.7, with no success. om -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540
![](https://secure.gravatar.com/avatar/b3407ff6ccd34c6e7c7a9fdcfba67a45.jpg?s=120&d=mm&r=g)
On Fri, Oct 03, 2003 at 11:18:28AM +0100, Oisin Mulvihill wrote:
Why are there two deferred objects returned by storeFile? The documentation says:
Try looking at twisted/test/test_ftp.py, line 251 (the testStor method of FTPClientAndServerTests). Hopefully that should help fill in the blanks. -Andrew.
![](https://secure.gravatar.com/avatar/4a58f201f6865057fdac810ad1481bcc.jpg?s=120&d=mm&r=g)
Andrew Bennetts wrote:
Hi, I know its been a while, but the project I needed this for has gone on the back burner. I was going to do an example for how I got the file to send and then do a recieve. I haven't got around to it. So for posterity and for anyone else who hits the same problem here is the modified ftp example that shows how I uploaded a file using twisted. The answer was in the unit tests I'd just forgotten twisted had them. All the best, om -- Oisin Mulvihill Engines Of Creation Email: oisin@enginesofcreation.ie Work: +353 1 6791602 Mobile: +353 868191540 # Twisted, the Framework of Your Internet # Copyright (C) 2001 Matthew W. Lefkowitz # # This library is free software; you can redistribute it and/or # modify it under the terms of version 2.1 of the GNU Lesser General Public # License as published by the Free Software Foundation. # # This library is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU # Lesser General Public License for more details. # # You should have received a copy of the GNU Lesser General Public # License along with this library; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA """ An example of using the FTP client """ # Twisted imports from twisted.protocols.ftp import FTPClient, FTPFileListProtocol from twisted.internet.protocol import Protocol, ClientCreator from twisted.protocols.basic import FileSender from twisted.python import usage from twisted.internet import reactor # Standard library imports import string import sys try: from cStringIO import StringIO except ImportError: from StringIO import StringIO # Define some callbacks def success(response): print 'Success! Got response:' print '---' if response is None: print None else: print string.join(response, '\n') print '---' def fail(error): print 'Failed. Error was:' print error from twisted.internet import reactor reactor.stop() class Options(usage.Options): optParameters = [['host', 'h', ''], ['port', 'p', 1221], ['username', 'u', 'anonymous'], ['password', None, 'anonymous@example.com'], ['passive', None, 0], ['debug', 'd', 1], ] def run(): # Get config config = Options() config.parseOptions() config.opts['port'] = int(config.opts['port']) config.opts['passive'] = int(config.opts['passive']) config.opts['debug'] = int(config.opts['debug']) # Create the client FTPClient.debug = config.opts['debug'] creator = ClientCreator(reactor, FTPClient, config.opts['username'], config.opts['password'], passive=config.opts['passive']) creator.connectTCP(config.opts['host'], config.opts['port']).addCallback(connectionMade) print "**** FTP start -" reactor.run() print "- FTP stop ****" def sendfile(consumer, fileObj): """Its now time to send the file, so read and write it calling finish when we've finished. """ print "2.1 consumer, fileObject -", consumer, fileObj consumer.write(fileObj.read()) print "2.2" consumer.finish() print "2.3" reactor.stop() print "2.4 Exit Time." def connectionMade(ftpClient): """Ok, we're connected to the server. """ try: print "1.0" my_file = StringIO("Hello! Did this get sent to the remote side?") print "1.1" my_filename = 'myfile.txt' print "1.2" ftpClient.pwd().addCallbacks(success, fail) print "1.3" dC, dL = ftpClient.storeFile(my_filename) print "1.4" dC.addCallback(sendfile, my_file) print "1.5" dL.addCallbacks(success, fail) print "1.6" except: print "Exception -", sys.exc_value reactor.stop() # this only runs if the module was *not* imported if __name__ == '__main__': run()
participants (3)
-
Andrew Bennetts
-
Jp Calderone
-
Oisin Mulvihill