ftplib limitations?

[Stefan Schwarzer]

Hi durumdara,

On 2010-08-25 09:43, durumdara wrote:
>> I can imagine the error message (a full traceback if
>> possible) would help to say a bit more about the cause of
>> the problem and maybe what to do about it.
> 
> This was:
> 
> Filename: "Repositories 20100824_101805 (Teljes).zip" Size: 1530296127
> ..download: 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
> Traceback (most recent call last):
>   File "C:\D\LocalBackup\ftpdown.py", line 31, in <module>
>     ftp.retrbinary("retr " + s, CallBack)
>   File "C:\Python26\lib\ftplib.py", line 401, in retrbinary
>     return self.voidresp()
>   File "C:\Python26\lib\ftplib.py", line 223, in voidresp
>     resp = self.getresp()
>   File "C:\Python26\lib\ftplib.py", line 209, in getresp
>     resp = self.getmultiline()
>   File "C:\Python26\lib\ftplib.py", line 195, in getmultiline
>     line = self.getline()
>   File "C:\Python26\lib\ftplib.py", line 182, in getline
>     line = self.file.readline()
>   File "C:\Python26\lib\socket.py", line 406, in readline
>     data = self._sock.recv(self._rbufsize)
> socket.error: [Errno 10054] A lÚtez§ kapcsolatot a tßvoli ßllomßs
> kÚnyszerÝtette
> n bezßrta
> 
> So this message is meaning that the remote station forced close the
> existing connection.

The file transfer protocol uses two connections for data
transfers, a control connection to send commands and
responses, and a data connection for the data payload
itself.

Now it may be that the data connection, after having started
the transfer, works as it should, but the control connection
times out because the duration of the transfer is too long.
A hint at this is that the traceback above contains
`getline` and `readline` calls which strongly suggest that
this socket was involved in some text transfer (presumably
for a status message).

Most FTP servers are configured for a timeout of 5 or 10
minutes. If you find that the file transfers don't fail
reproducably for a certain size limit, it's probably not the
size of the file that causes the problem but some timing
issue (see above).

What to do about it? One approach is to try to get the
timeout value increased. Of course that depends on the
relation between you and the party running the server.
Another approach is to catch the exception and ignore it.
To make sure you only ignore timeout messages, you may want
to check the status code at the start of the error message
and re-raise the exception if it's not the status expected
for a timeout. Something along the lines of:

    try:
        # transer involving `retrbinary`
    except socket.error, exc:
        if str(exc).startswith("[Errno 10054] "):
            pass
        else:
            raise

Note, however, that this is a rather brittle way to handle
the problem, as the status code or format of the error
message may depend on the platform your program runs on,
library versions, etc.

In any case you should close and re-open the FTP connection
after you got the error from the server.

> Now I'm trying with saving the file into temporary file, not hold in
> memory.

If my theory holds, that shouldn't make a difference. But
maybe my theory is wrong. :)

Could you do me a favor and try your download with ftputil
[1]? The code should be something like:

    import ftputil

    host = ftputil.FTPHost(server, userid, passwd)
    for name in host.listdir(host.curdir):
        host.download(name, name, 'b')
    host.close()

There's neither a need nor - at the moment - a possibility
to specify a callback if you just want the download. (I'm
working on the callback support though!)

For finding the error, it's of course better to just use the
download command for the file that troubles you.

I'm the maintainer of ftputil and if you get the same or
similar error here, I may find a workaround for ftputil. As
it happens, someone reported a similar problem (_if_ it's
the same problem in your case) just a few days ago. [2]

[1] http://ftputil.sschwarzer.net
[2] http://www.mail-archive.com/ftputil@codespeak.net/msg00141.html

Stefan