waling a directory with very many files
python.list at tim.thechases.com
Sun Jun 14 22:07:25 EDT 2009
>> i can traverse a directory using os.listdir() or os.walk(). but if a
>> directory has a very large number of files, these methods produce very
>> large objects talking a lot of memory.
>> in other languages one can avoid generating such an object by walking
>> a directory as a liked list. for example, in c, perl or php one can
>> use opendir() and then repeatedly readdir() until getting to the end
>> of the file list. it seems this could be more efficient in some
>> is there a way to do this in python? i'm relatively new to the
>> language. i looked through the documentation and tried googling but
>> came up empty.
> You did not specify version. In Python3, os.walk has become a generater
> function. So, to answer your question, use 3.1.
Since at least 2.4, os.walk has itself been a generator.
However, the contents of the directory (the 3rd element of the
yielded tuple) is a list produced by listdir() instead of a
generator. Unless listdir() has been changed to a generator
instead of a list (which other respondents seem to indicate has
not been implemented), this doesn't address the OP's issue of
"lots of files in a single directory".
More information about the Python-list