[Python-3000] Removal of os.path.walk

[Nick Coghlan]

Martin v. Löwis wrote:
> Guido van Rossum wrote:
>> There is one use case I can see for an iterator-version of
>> os.listdir() (to be named os.opendir()): when globbing a huge
>> directory looking for a certain pattern. Using os.listdir() you end up
>> needed enough memory to hold all of the names at once. Using
>> os.opendir() you would need only enough memory to hold all of the
>> names THAT MATCH.
> 
> You would still have to read the entire directory, right?
> In that kind of class, there is a number of applications;
> e.g. du(1) also wouldn't have to create a list of all files
> in the directory, but add the sizes of the files incrementally.
> 
> So the question really is whether it is a problem to keep
> all file names in memory simultaneously. As Aahz says, the
> total memory consumption for a large directory is still
> comparatively low, for today's machines.

I think Giovanni's point is an important one as well - with an iterator, 
you can pipeline your operations far more efficiently, since you don't 
have to wait for the whole directory listing before doing anything (e.g. 
if you're doing some kind of move/rename operation on a directory, you 
can start copying the first file to its new location without having to 
wait for the directory read to finish).

Reducing the startup delays of an operation can be a very useful thing 
when it comes to providing a user with a good feeling of responsiveness 
from an application (and if it allows the application to more 
effectively pipeline something, there may be an actual genuine 
improvement in responsiveness, rather than just the appearance of one).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org