[Python-ideas] Speed up os.walk() 5x to 9x by using file attributes from FindFirst/NextFile() and readdir()

Nick Coghlan ncoghlan at gmail.com
Wed Nov 14 10:33:03 CET 2012


On Wed, Nov 14, 2012 at 5:14 PM, Ronald Oussoren <ronaldoussoren at mac.com>wrote:

> How did you measure the 5x speedup you saw with you modified os.walk?
>
> It would be interesting to see if Unix platforms have a simular speedup,
> because
> if they don't the new API could just return the results of stat (or lstat
> ...).
>
>
One thing to keep in mind with these kind of metrics is that I/O latency is
a major factor. Solid state vs spinning disk vs network drive is going to
make a *big* difference to the relative performance of the different
mechanisms. With NFS (et al), it's particularly important to minimise the
number of round trips to the server (that's why the new dir listing caching
in the 3.3 import system results in such dramatic speed-ups when some of
the sys.path entries are located on network drives).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20121114/257a1a03/attachment.html>


More information about the Python-ideas mailing list