[Python-ideas] os.listdir iteration support
gnewsg at gmail.com
Fri Nov 23 22:26:40 CET 2007
On 23 Nov, 21:23, "Guido van Rossum" <gu... at python.org> wrote:
> But how many FTP servers are written in Python *and* have directories
> with 20,000 files in them?
I sincerely don't know.
Surely it's a rather specific use case, but it is one of the tasks
which takes the longest amount of time on an FTP server. 20,000 is
probably an exaggerated hypothetical situation, so I did a simple test
with a more realistic scenario.
On windows a very crowded directory is C:\windows\system32. Currently
the C:\windows\system32 of my Windows XP workstation contains 2201
I tried to run the code below which is how an FTP server should
properly respond to a "LIST" command issued by client.
It took 1.70300006866 seconds to complete the first time and
0.266000032425 the second one.
I don't know if such specific use case could justify a listdir
generators support to have into the stdlib but having something like
Greg Ewing's opendirs module could have saved a lot of time in this
import os, stat, time
from tarfile import filemode
import pwd, grp
pwd = grp = None
"""Return a directory listing emulating "/bin/ls -lA" UNIX
This is how output appears to client:
-rw-rw-rw- 1 owner group 7045120 Sep 02 3:47 music.mp3
drwxrwxrwx 1 owner group 0 Aug 31 18:50 e-books
-rw-rw-rw- 1 owner group 380 Sep 02 3:40 module.py
listing = os.listdir(directory)
result = 
for basename in listing:
file = os.path.join(directory, basename)
# if the file is a broken symlink, use lstat to get stat for
# the link
stat_result = os.stat(file)
stat_result = os.lstat(file)
perms = filemode(stat_result.st_mode) # permissions
nlinks = stat_result.st_nlink # number of links to inode
if not nlinks: # non-posix system, let's use a bogus value
nlinks = 1
if pwd and grp:
# get user and group name, else just use the raw uid/gid
uname = pwd.getpwuid(stat_result.st_uid).pw_name
uname = stat_result.st_uid
gname = grp.getgrgid(stat_result.st_gid).gr_name
gname = stat_result.st_gid
# on non-posix systems the only chance we use default
# bogus values for owner and group
uname = "owner"
gname = "group"
size = stat_result.st_size # file size
# stat.st_mtime could fail (-1) if file's last modification
# time is too old, in that case we return local time as last
# modification time.
mtime = time.strftime("%b %d %H:%M",
mtime = time.strftime("%b %d %H:%M")
# if the file is a symlink, resolve it, e.g. "symlink ->
basename = basename + " -> " + os.readlink(file)
# formatting is matched with proftpd ls output
result.append("%s %3s %-8s %-8s %8s %s %s\r\n" %(
perms, nlinks, uname, gname, size, mtime, basename))
if __name__ == '__main__':
before = time.time()
print time.time() - before
More information about the Python-ideas