Iterating over files of a huge directory
tjreedy at udel.edu
Mon Dec 17 22:27:22 CET 2012
On 12/17/2012 10:28 AM, Gilles Lenfant wrote:
> I have googled but did not find an efficient solution to my problem.
> My customer provides a directory with a huuuuge list of files (flat,
> potentially 100000+) and I cannot reasonably use
> os.listdir(this_path) unless creating a big memory footprint.
Is is really big enough to be a real problem? See below.
> So I'm looking for an iterator that yields the file names of a
> directory and does not make a giant list of what's in.
> i.e :
> for filename in enumerate_files(some_directory): # My cooking...
As I said there, I personally think (and still do) that listdir should
have been changed in 3.0 to return an iterator rather than a list.
Developers who count more than me disagree on the basis that no
application has the millions of directory entries needed to make space a
real issue. They also claim that time is a wash either way.
As for space, 100000 entries x 100 bytes/entry (generous guess at
average) = 10,000,000 bytes, no big deal with gigabyte memories. So the
logic goes. A smaller example from my machine with 3.3.
from sys import getsizeof
"Get size of flat sequence and contents"
return sum((getsizeof(item) for item in seq), getsizeof(seq))
d = os.listdir()
print(seqsize([1,2,3]), len(d), seqsize(d))
172 45 3128
The size per entry is relatively short because the two-level directory
prefix for each path is only about 15 bytes. By using 3.3 rather than
3.0-3.2, the all-ascii-char unicode paths only take 1 byte per char
rather than 2 or 4.
If you disagree with the responses on the issue, after reading them,
post one yourself with real numbers.
Terry Jan Reedy
More information about the Python-list