[Python-ideas] Speed up os.walk() 5x to 9x by using file attributes from FindFirst/NextFile() and readdir()

[Steve Dower]

Robert Collins wrote:
>On Wed, Nov 14, 2012 at 11:52 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>  - chdir to the directory before you stat and use a relative path: it
>>> turns out when working with many files that the overhead of absolute
>>> paths is substantial.
>>
>> How about using fstatat() instead? chdir() is a no-no since it's
>> a process-wide setting.
>
> fstatat looks *perfect*. Thanks for the pointer.
>
> I forget the win32 behaviour, but on Linux a thread is a process ->
> chdir is localised to the process and not altered across threads.

chdir is a very bad idea for libraries on Windows - it is meant mainly for the user and not the application. It has also been removed completely for Windows 8 apps (not desktop applications, just the new style ones).

Full paths may involve some overhead, but at least with FindFirst/NextFile this should be limited to memory copies and not file lookups.

Cheers,
Steve