waling a directory with very many files

[Steven D'Aprano]

On Sun, 14 Jun 2009 22:35:50 +0200, Andre Engels wrote:

> On Sun, Jun 14, 2009 at 6:35 PM, tom<fsb at thefsb.org> wrote:
>> i can traverse a directory using os.listdir() or os.walk(). but if a
>> directory has a very large number of files, these methods produce very
>> large objects talking a lot of memory.
>>
>> in other languages one can avoid generating such an object by walking a
>> directory as a liked list. for example, in c, perl or php one can use
>> opendir() and then repeatedly readdir() until getting to the end of the
>> file list. it seems this could be more efficient in some applications.
>>
>> is there a way to do this in python? i'm relatively new to the
>> language. i looked through the documentation and tried googling but
>> came up empty.
> 
> What kind of directories are those that just a list of files would
> result in a "very large" object? I don't think I have ever seen
> directories with more than a few thousand files...


You haven't looked very hard :)

$ pwd
/home/steve/.thumbnails/normal
$ ls | wc -l
33956

And I periodically delete thumbnails, to prevent the number of files 
growing to hundreds of thousands.



-- 
Steven