waling a directory with very many files

[MRAB]

Christian Heimes wrote:
> tom schrieb:
>> i can traverse a directory using os.listdir() or os.walk(). but if a
>> directory has a very large number of files, these methods produce very
>> large objects talking a lot of memory.
>>
>> in other languages one can avoid generating such an object by walking
>> a directory as a liked list. for example, in c, perl or php one can
>> use opendir() and then repeatedly readdir() until getting to the end
>> of the file list. it seems this could be more efficient in some
>> applications.
>>
>> is there a way to do this in python? i'm relatively new to the
>> language. i looked through the documentation and tried googling but
>> came up empty.
> 
> Some time ago we had a discussion about turning os.listdir() into a
> generator. No conclusion was agreed on. We also thought about exposing
> the functions opendir(), readdir(), closedir() and friends but as far as
> I know and as far as I've checked the C code in Modules/posixmodule.c
> none of the functions as been added.
> 
Perhaps if there's a generator it should be called iterdir(). Or would
it be unPythonic to have listdir() and iterdir()? Probably.

> For now you are on your own to implement wrappers for the system calls.
> For the distant future you may see the appropriate functions in the os
> module. A mail to the python ideas list may increase your chances. ;)
>