I want to port my ftp client class from standard library (ftplib) to twisted ftpclient. I start to work with ftpclient example and with a "standard" ftp server, it work well, of course :), but I'm working on a not standard server (hylafax), that not accept the standard commands, so now with the standard library, every time that I want to communicate with it, I send its internal commands. Is it possible to "translate" my very short and simple code to twisted ftpclint? However, I have already tried to modify the simple generic client example "EchoClient", and it work until I make a "LIST status" request, that it would have to return a short string (100-150 characters), because if I use passive mode (command 'PASV') I receive a 425 error, and if I don't use it, I don't receive anything (I wait...). Can someone point me to the right direction ? Thanks, Michele
On Wed, Aug 03, 2005 at 12:22:59PM +0200, Michele Petrazzo wrote:
Probably. If you need to send non-standard commands, you can do that with ftpclient.queueStringCommand("BLAH blah blah").
The right way is to re-use the existing FTPClient logic as much as possible, rather than reimplementing it. What specific problems are you having with twisted.protocols.ftp.FTPClient? -Andrew.
Andrew Bennetts wrote:
<-cut->
I change the ftpclient example with this code: def run(): # Create the client creator = ClientCreator(reactor, FTPClient, user, passwd, 0) creator.connectTCP('192.168.1.1', 4559).addCallback(connectionMade) reactor.run() def connectionMade(ftpClient): # Get the current working directory ftpClient.pwd().addCallbacks(success, fail) One problem is that if the user has no password, I receive an error [1] (if I try to connect with a simple ftp client, when I connect to the server, if it see that I have no password, it connect me without password request), however if the user _has_ password, the client connect correctly. Second: if the user _has_ password, so I can connect, I want to send "LIST status" command, if I use: ftpClient.queueStringCommand("LIST").addCallbacks(success, fail) I receive [2], and if I use: fileList = FTPFileListProtocol() d = ftpClient.list(' status ', fileList) d.addCallbacks(showFiles, fail, callbackArgs=(fileList,)) I receive [3] Returned data: [1] Failed. Error was: [Failure instance: Traceback (failure with no frames): twisted.protocols.ftp.ConnectionLost: ('FTP connection lost', <twisted.python.failure.Failure twisted.protocols.ftp.CommandFailed>) [2] Success! Got response: --- 257 "/" is the current directory. --- Failed. Error was: [Failure instance: Traceback (failure with no frames): twisted.protocols.ftp.CommandFailed: ['425 Cannot build data connection: Connection refused.'] [3] Success! Got response: --- 257 "/" is the current directory. --- Failed. Error was: [Failure instance: Traceback (failure with no frames): <type 'tuple'>: (<twisted.python.failure.Failure <type 'tuple'>>, 0) ]
-Andrew.
Thanks, Michele
On Wed, Aug 03, 2005 at 04:53:34PM +0200, Michele Petrazzo wrote:
Before connecting, can you add: FTPClient.debug = True Then we can see a full transcript of what's going on.
Right. LIST requires a data connection to be established, and directly issuing a LIST without arranging for a PASV or PORT command and associated connection will fail.
Can you retry this with latest Twisted from SVN? There's a bug in error reporting and DeferredLists here that's been fixed in SVN that will make this much clearer. I don't understand why you're passing ' status ' instead of just 'status', though. -Andrew.
Andrew Bennetts wrote:
I add that line, but I can't see any new line except that I already seen
Ok, but what I do with to make it work?
Yes. Just update the my two (win and lin) machines. [1]
I don't understand why you're passing ' status ' instead of just 'status', though.
But, I don't know :) With this change " status " -> "status" now I don't receive any error! But what I receive is an empty file list [2] (on all status and recvq sub-directory), both case if I change directory with the cwd command and call ftpClient.list(".") or, call it with "status/"-"recvq/". But I know that into that sub-directory there are a lot of files... [1] The return are (on both OS): [Failure instance: Traceback (failure with no frames): twisted.internet.defer.FirstError: FirstError(<twisted.python.failure.Failure twisted.internet.defer.FirstError>, 0) ] [2] Processed file listing: Total: 0 files Thanks, Michele
On Wed, Aug 03, 2005 at 07:23:56PM +0200, Michele Petrazzo wrote:
Odd. It should log everything.
Use FTPClient.list ;) Seriously, look at the implementation of FTPClient.list, it's very simple: def list(self, path, protocol): # ...docstring elided... if path is None: path = '' return self.receiveFromConnection(['LIST ' + self.escapePath(path)], protocol) FTPClient.receiveFromConnection is a helper function that Does The Right Thing depending on if self.passive is set or not. [...]
Ok, I suspect what was happening was that the server was returning an error like "' status ' does not exist". The error reporting from FTPClient is pretty messy, unfortunately.
Perhaps FTPFileListProtocol can't parse your file listing -- it tries to parse stuff more-or-less like: -rw-r--r-- 1 root other 531 Jan 29 03:26 README If you have another format, you'll need to write your own. As a starting point, you could try: class LineRecorder(basic.LineReceiver): def connectionMade(self): self.lines = [] def lineReceived(self,line): self.lines.append(line) Alternatively, subclass FTPFileListProtocol and override the unknownLine method.
Sorry, I was too hasty here -- you also need to change your errback to print failure.subFailure or perhaps failure.subFailure.subFailure rather than just failure. There are several layers of failure here (because of several layers of DeferredLists), which makes it ugly. FTPClient shouldn't be exposing the implementation detail of DeferredLists like this, but at the moment it does. -Andrew.
Andrew Bennetts wrote:
Strange. Do you want that I change something to understand the problem? This happen also with the demo "ftpclient.py"!
Ok, but what I do with to make it work?
Use FTPClient.list ;)
Obviously! :)
But what I receive is an empty file list
With unknownLine overwritten I receive all the data correctly. What the are more right way (the advice method) to proceed? Simple override that method or create a new HylafaxFTPFileListProtocol ?
After some code tries, I receive this response: user login that has no password : [1] user login with password, list of not exist directory: [2] Like you can see, I must find the return errors inside the failure instance at two differents point (I believe because, like you wrote, there are several layers). You think that this "problem" will be fixed in the future, for make the error more simple to understand? I attach a diff file that solve my problem with the empty password and the "PASS" command. For me it work well. I'll very happy if you include my patch! [1] code: def fail(error): print error.value[1] [Failure instance: Traceback (failure with no frames): twisted.protocols.ftp.CommandFailed: ["500 'PASS ': Syntax error, expecting password."] ] [1] code: def fail(error): print error.value[0].value.subFailure test_new_hylaproto.py:49: DeprecationWarning: FirstError.__getitem__ is deprecated. Use attributes instead. [Failure instance: Traceback (failure with no frames): twisted.protocols.ftp.CommandFailed: ['550 pippo: No such file or directory.'] Thanks, Michele
On Thu, Aug 04, 2005 at 11:30:43AM +0200, Michele Petrazzo wrote:
If you can figure out why, I'd be happy, but I don't have time to investigate right now (I'm quite busy atm, which is why my replies are so brief, sorry).
Create a new HylafaxFTPFileListProtocol would be simplest, I think.
Ok, so your FTP server requires a password.
user login with password, list of not exist directory: [2]
Great, thanks for that output.
Yes, it will, I'll file a bug on it so that I don't forget. Thanks for uncovering it!
Interesting. Please file this at http://twistedmatrix.com/bugs, assign to spiv. Also, please use unified diffs ("diff -u") if you can. I think perhaps passing password=None might be a better way than sendPass=False, I'll think about it. Thank you very much! -Andrew.
Andrew Bennetts wrote:
On Thu, Aug 04, 2005 at 11:30:43AM +0200, Michele Petrazzo wrote:
I'll try next week. Now I have to modify my code to make the HylafaxFTPFileListProtocol work! :)
Ok, I'll try
No, the server don't required a password, so if the ftp library send the "PASS" command, it "anger".
Done, issue 1130
Thank you to you for all the help.
-Andrew.
Michele
On Wed, Aug 03, 2005 at 12:22:59PM +0200, Michele Petrazzo wrote:
Probably. If you need to send non-standard commands, you can do that with ftpclient.queueStringCommand("BLAH blah blah").
The right way is to re-use the existing FTPClient logic as much as possible, rather than reimplementing it. What specific problems are you having with twisted.protocols.ftp.FTPClient? -Andrew.
Andrew Bennetts wrote:
<-cut->
I change the ftpclient example with this code: def run(): # Create the client creator = ClientCreator(reactor, FTPClient, user, passwd, 0) creator.connectTCP('192.168.1.1', 4559).addCallback(connectionMade) reactor.run() def connectionMade(ftpClient): # Get the current working directory ftpClient.pwd().addCallbacks(success, fail) One problem is that if the user has no password, I receive an error [1] (if I try to connect with a simple ftp client, when I connect to the server, if it see that I have no password, it connect me without password request), however if the user _has_ password, the client connect correctly. Second: if the user _has_ password, so I can connect, I want to send "LIST status" command, if I use: ftpClient.queueStringCommand("LIST").addCallbacks(success, fail) I receive [2], and if I use: fileList = FTPFileListProtocol() d = ftpClient.list(' status ', fileList) d.addCallbacks(showFiles, fail, callbackArgs=(fileList,)) I receive [3] Returned data: [1] Failed. Error was: [Failure instance: Traceback (failure with no frames): twisted.protocols.ftp.ConnectionLost: ('FTP connection lost', <twisted.python.failure.Failure twisted.protocols.ftp.CommandFailed>) [2] Success! Got response: --- 257 "/" is the current directory. --- Failed. Error was: [Failure instance: Traceback (failure with no frames): twisted.protocols.ftp.CommandFailed: ['425 Cannot build data connection: Connection refused.'] [3] Success! Got response: --- 257 "/" is the current directory. --- Failed. Error was: [Failure instance: Traceback (failure with no frames): <type 'tuple'>: (<twisted.python.failure.Failure <type 'tuple'>>, 0) ]
-Andrew.
Thanks, Michele
On Wed, Aug 03, 2005 at 04:53:34PM +0200, Michele Petrazzo wrote:
Before connecting, can you add: FTPClient.debug = True Then we can see a full transcript of what's going on.
Right. LIST requires a data connection to be established, and directly issuing a LIST without arranging for a PASV or PORT command and associated connection will fail.
Can you retry this with latest Twisted from SVN? There's a bug in error reporting and DeferredLists here that's been fixed in SVN that will make this much clearer. I don't understand why you're passing ' status ' instead of just 'status', though. -Andrew.
Andrew Bennetts wrote:
I add that line, but I can't see any new line except that I already seen
Ok, but what I do with to make it work?
Yes. Just update the my two (win and lin) machines. [1]
I don't understand why you're passing ' status ' instead of just 'status', though.
But, I don't know :) With this change " status " -> "status" now I don't receive any error! But what I receive is an empty file list [2] (on all status and recvq sub-directory), both case if I change directory with the cwd command and call ftpClient.list(".") or, call it with "status/"-"recvq/". But I know that into that sub-directory there are a lot of files... [1] The return are (on both OS): [Failure instance: Traceback (failure with no frames): twisted.internet.defer.FirstError: FirstError(<twisted.python.failure.Failure twisted.internet.defer.FirstError>, 0) ] [2] Processed file listing: Total: 0 files Thanks, Michele
On Wed, Aug 03, 2005 at 07:23:56PM +0200, Michele Petrazzo wrote:
Odd. It should log everything.
Use FTPClient.list ;) Seriously, look at the implementation of FTPClient.list, it's very simple: def list(self, path, protocol): # ...docstring elided... if path is None: path = '' return self.receiveFromConnection(['LIST ' + self.escapePath(path)], protocol) FTPClient.receiveFromConnection is a helper function that Does The Right Thing depending on if self.passive is set or not. [...]
Ok, I suspect what was happening was that the server was returning an error like "' status ' does not exist". The error reporting from FTPClient is pretty messy, unfortunately.
Perhaps FTPFileListProtocol can't parse your file listing -- it tries to parse stuff more-or-less like: -rw-r--r-- 1 root other 531 Jan 29 03:26 README If you have another format, you'll need to write your own. As a starting point, you could try: class LineRecorder(basic.LineReceiver): def connectionMade(self): self.lines = [] def lineReceived(self,line): self.lines.append(line) Alternatively, subclass FTPFileListProtocol and override the unknownLine method.
Sorry, I was too hasty here -- you also need to change your errback to print failure.subFailure or perhaps failure.subFailure.subFailure rather than just failure. There are several layers of failure here (because of several layers of DeferredLists), which makes it ugly. FTPClient shouldn't be exposing the implementation detail of DeferredLists like this, but at the moment it does. -Andrew.
Andrew Bennetts wrote:
Strange. Do you want that I change something to understand the problem? This happen also with the demo "ftpclient.py"!
Ok, but what I do with to make it work?
Use FTPClient.list ;)
Obviously! :)
But what I receive is an empty file list
With unknownLine overwritten I receive all the data correctly. What the are more right way (the advice method) to proceed? Simple override that method or create a new HylafaxFTPFileListProtocol ?
After some code tries, I receive this response: user login that has no password : [1] user login with password, list of not exist directory: [2] Like you can see, I must find the return errors inside the failure instance at two differents point (I believe because, like you wrote, there are several layers). You think that this "problem" will be fixed in the future, for make the error more simple to understand? I attach a diff file that solve my problem with the empty password and the "PASS" command. For me it work well. I'll very happy if you include my patch! [1] code: def fail(error): print error.value[1] [Failure instance: Traceback (failure with no frames): twisted.protocols.ftp.CommandFailed: ["500 'PASS ': Syntax error, expecting password."] ] [1] code: def fail(error): print error.value[0].value.subFailure test_new_hylaproto.py:49: DeprecationWarning: FirstError.__getitem__ is deprecated. Use attributes instead. [Failure instance: Traceback (failure with no frames): twisted.protocols.ftp.CommandFailed: ['550 pippo: No such file or directory.'] Thanks, Michele
On Thu, Aug 04, 2005 at 11:30:43AM +0200, Michele Petrazzo wrote:
If you can figure out why, I'd be happy, but I don't have time to investigate right now (I'm quite busy atm, which is why my replies are so brief, sorry).
Create a new HylafaxFTPFileListProtocol would be simplest, I think.
Ok, so your FTP server requires a password.
user login with password, list of not exist directory: [2]
Great, thanks for that output.
Yes, it will, I'll file a bug on it so that I don't forget. Thanks for uncovering it!
Interesting. Please file this at http://twistedmatrix.com/bugs, assign to spiv. Also, please use unified diffs ("diff -u") if you can. I think perhaps passing password=None might be a better way than sendPass=False, I'll think about it. Thank you very much! -Andrew.
Andrew Bennetts wrote:
On Thu, Aug 04, 2005 at 11:30:43AM +0200, Michele Petrazzo wrote:
I'll try next week. Now I have to modify my code to make the HylafaxFTPFileListProtocol work! :)
Ok, I'll try
No, the server don't required a password, so if the ftp library send the "PASS" command, it "anger".
Done, issue 1130
Thank you to you for all the help.
-Andrew.
Michele
participants (2)
-
Andrew Bennetts -
Michele Petrazzo