[Python-ideas] Speed up os.walk() 5x to 9x by using file attributes from FindFirst/NextFile() and readdir()
Steve.Dower at microsoft.com
Wed Nov 14 20:04:18 CET 2012
Robert Collins wrote:
>On Wed, Nov 14, 2012 at 11:52 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>> - chdir to the directory before you stat and use a relative path: it
>>> turns out when working with many files that the overhead of absolute
>>> paths is substantial.
>> How about using fstatat() instead? chdir() is a no-no since it's
>> a process-wide setting.
> fstatat looks *perfect*. Thanks for the pointer.
> I forget the win32 behaviour, but on Linux a thread is a process ->
> chdir is localised to the process and not altered across threads.
chdir is a very bad idea for libraries on Windows - it is meant mainly for the user and not the application. It has also been removed completely for Windows 8 apps (not desktop applications, just the new style ones).
Full paths may involve some overhead, but at least with FindFirst/NextFile this should be limited to memory copies and not file lookups.
More information about the Python-ideas