[Python-bugs-list] [ python-Bugs-751758 ] ftplib.retrbinary fails when called from retrlines callback

Sat, 14 Jun 2003 17:45:14 -0700

Bugs item #751758, was opened at 2003-06-10 01:51
Message generated for change (Comment added) made by sjones
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=751758&group_id=5470

Category: Python Library
Group: Python 2.2.2
Status: Open
Resolution: None
Priority: 5
Submitted By: Christian Long (christianmlong)
Assigned to: Nobody/Anonymous (nobody)
Summary: ftplib.retrbinary fails when called from retrlines callback

Initial Comment:

Subject: ftplib.retrbinary() fails when called from
inside retrlines() callback function

I'm using ftplib to backup files from a Linux server to
a Windows 2000 worksation. 

Client:  
Windows 2000 Pro
ActivePython 2.2.2 Build 224 (ActiveState Corp.) based on
Python 2.2.2 (#37, Nov 26 2002, 10:24:37) [MSC 32 bit
(Intel)] on win32
Komodo IDE

Server:
ProFTP server (ProFTPd version 1.25) 
Mandrake Linux 9.0

Summary:

When I use it like this it works fine.

# Build a list of files that are on the remote server
f.retrlines('NLST', makeListOfFiles)

--then--
# Iterate over the list, retrieving each file
for remoteFileName in listOfFiles:
    --snip--
    f.retrbinary('RETR %s' % remoteFileName,
localFile.write)
    --snip--

But it fails if I try to do the retrieve directly in my
callback function.

def transferFile(listLine):
    --snip--
    f.retrbinary('RETR %s' % remoteFileName,
localFile.write)  <--fails here on first time through
    --snip--

# get list of files from server, adn transfer each file
as it gets listed
f.retrlines('LIST', transferFile)

--snip--
  File "D:\My Documents\Computer World\Python
Resources\My Utilities\backup_remote_files.py", line
45, in ?
    f.retrlines('LIST', transferFile)
  File "C:\Python22\lib\ftplib.py", line 413, in retrlines
    callback(line)
  File "D:\My Documents\Computer World\Python
Resources\My Utilities\backup_remote_files.py", line
36, in transferFile
    f.retrbinary('RETR mra.py', localFile.write)
--snip--
  File "C:\Python22\lib\ftplib.py", line 300, in makepasv
    host, port = parse227(self.sendcmd('PASV'))
  File "C:\Python22\lib\ftplib.py", line 572, in parse227
    raise error_reply, resp
error_reply: 200 Type set to I.

It looks like the server is returning a 200 instead of
a 227 when retrbinary() is called inside a callback
function for retrlines().

Files:

2 Files are included:  a broken version and a version
that works

This One Is Broken -  retrbinary() called from inside a
callback function for retrlines().
===================================================
import ftplib
import os
import time

REMOTE_DIR = "/home/mydir"
LOCAL_DIR = "C:\My Documents"
TIME_FORMAT = "%y%m%d"    # YYMMDD, like 030522

def transferFile(listLine):
    # Strips the file name from a line of a
    # directory listing, and gets it from the
    # server.  Depends on filenames
    # with no embedded spaces or extra dots.
    if listLine.endswith('.py'):
        #Split file name on the dot
        splitFileName=remoteFileName.split('.')
        # Add a timestamp
        localFileName="%s_%s.%s" % (splitFileName[0],

time.strftime(TIME_FORMAT),
                                    splitFileName[1])
        # Open a local file for (over)writing, in
binary mode.
        # print os.path.join(LOCAL_DIR,localFileName)

localFile=file(os.path.join(LOCAL_DIR,localFileName), 'wb')
        print remoteFileName
        print localFile
        # Execute the FTP retrieve command, calling
        # the write() function of the local file
        # for each block retrieved from the FTP server
        # BUG: This should work, but I get the
following traceback
        f.retrbinary('RETR %s' % remoteFileName,
localFile.write)    #<--- Fails

              #     Here

#        mra.py
#<open file 'D:\My
Documents\Work\IA\Miller\MRA\Dev\Backup of remote
files\mra_030610.py', mode 'wb' at 0x00886B70>
#Traceback (most recent call last):
#  File "C:\Program Files\ActiveState Komodo
2.3\callkomodo\kdb.py", line 430, in _do_start
#    self.kdb.run(code_ob, locals, locals)
#  File "C:\Python22\lib\bdb.py", line 349, in run
#    exec cmd in globals, locals
#  File "D:\My Documents\Computer World\Python
Resources\My Utilities\backup_remote_files.py", line
45, in ?
#    f.retrlines('LIST', transferFile)
#  File "C:\Python22\lib\ftplib.py", line 413, in retrlines
#    callback(line)
#  File "D:\My Documents\Computer World\Python
Resources\My Utilities\backup_remote_files.py", line
36, in transferFile
#    f.retrbinary('RETR mra.py', localFile.write)
#  File "C:\Python22\lib\ftplib.py", line 385, in
retrbinary
#    conn = self.transfercmd(cmd, rest)
#  File "C:\Python22\lib\ftplib.py", line 346, in
transfercmd
#    return self.ntransfercmd(cmd, rest)[0]
#  File "C:\Python22\lib\ftplib.py", line 322, in
ntransfercmd
#    host, port = self.makepasv()
#  File "C:\Python22\lib\ftplib.py", line 300, in makepasv
#    host, port = parse227(self.sendcmd('PASV'))
#  File "C:\Python22\lib\ftplib.py", line 572, in parse227
#    raise error_reply, resp
#error_reply: 200 Type set to I.

# The problem is that the self.sendcmd('PASV') call is
not getting a 227
# reply from the server.  Rather, it is getting a 200
reply, confirming
# that the type was set to I (Image).

        localFile.flush()
        localFile.close()

f=ftplib.FTP('server', 'user', 'password')
f.cwd(REMOTE_DIR)
# List directory contents, and call the transferFile
# function on each line in the listing
f.retrlines('LIST', transferFile)

f.close()

===================================================

This One Works - retlines() builds a list, and then
files are transferred by iterating over
that list and calling retrbinary() for each.
===================================================
import ftplib
import os
import time

REMOTE_DIR = "/home/mydir"
LOCAL_DIR = "C:\My Documents"
TIME_FORMAT = "%y%m%d"    # YYMMDD, like 030522

listOfFiles = []

def makeListOfFiles(remoteFileName):
    # Strips the file name from a line of a
    # directory listing, and gets file from the
    # server.  Depends on filenames
    # with no embedded spaces or extra dots.
    if remoteFileName.endswith('.py'):
        listOfFiles.append(remoteFileName)

f=ftplib.FTP('server', 'user', 'password')
f.cwd(REMOTE_DIR)
# List directory contents, and call the transferFile
# function on each line in the listing
f.retrlines('NLST', makeListOfFiles)
print listOfFiles

for remoteFileName in listOfFiles:
    #Split file name on the dot
    splitFileName=remoteFileName.split('.')
    # Add a timestamp
    localFileName="%s_%s.%s" % (splitFileName[0],
                                time.strftime(TIME_FORMAT),
                                splitFileName[1])
    # Open a local file for (over)writing, in binary mode.
    # print os.path.join(LOCAL_DIR,localFileName)

localFile=file(os.path.join(LOCAL_DIR,localFileName), 'wb')

    # Execute the FTP retrieve command, calling
    # the write() function of the local file
    # for each block retrieved from the FTP server
    f.retrbinary('RETR %s' % remoteFileName,
localFile.write)
    localFile.flush()
    localFile.close()

f.close()

===================================================

----------------------------------------------------------------------

Comment By: Shannon Jones (sjones)
Date: 2003-06-14 19:45

Message:
Logged In: YES 
user_id=589306

The problem seems to happen when you use a callback within a
function that was called as a callback. Here is a much
simpler case that demonstrates the problem:

-----------------------------
import ftplib

def transferFile(listLine):
    filename = listLine.split()[-1]
    if filename == 'README':
        # Note that retrlines uses a default
        # callback that just prints the file
        f.retrlines('RETR README') # <-- Fails

f=ftplib.FTP('ftp.python.org', 'ftp', 'anon@')
f.cwd('/pub/python')
f.retrlines('LIST', transferFile)
f.close()
-----------------------------

This fails with the following:

Traceback (most recent call last):
  File "ftptest.py", line 10, in ?
    f.retrlines('LIST', transferFile)
  File "/home/sjones/src/python/dist/src/Lib/ftplib.py",
line 407, in retrlines
    callback(line)
  File "ftptest.py", line 6, in transferFile
    f.retrlines('RETR README') # <-- Fails
  File "/home/sjones/src/python/dist/src/Lib/ftplib.py",
line 396, in retrlines
    conn = self.transfercmd(cmd)
  File "/home/sjones/src/python/dist/src/Lib/ftplib.py",
line 345, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File "/home/sjones/src/python/dist/src/Lib/ftplib.py",
line 321, in ntransfercmd
    host, port = self.makepasv()
  File "/home/sjones/src/python/dist/src/Lib/ftplib.py",
line 299, in makepasv
    host, port = parse227(self.sendcmd('PASV'))
  File "/home/sjones/src/python/dist/src/Lib/ftplib.py",
line 566, in parse227
    raise error_reply, resp
ftplib.error_reply: 200 Type set to A.

Note this is with the current CVS version on Redhat 9.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=751758&group_id=5470