[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Ben Hoyt]

Ben Hoyt added the comment:

Thanks. I thought about that -- but I think I *want* to benchmark it when
they're cached, so that we're comparing apples with apples, cached system
calls with cached systems calls. The benchmark would almost certainly be a
lot "better" (BetterWalk would be even faster) if I was comparing the
non-cached results. I'll think about it some more though.

Thoughts?

-Ben

On Fri, May 3, 2013 at 7:03 PM, Charles-François Natali <
report at bugs.python.org> wrote:

>
> Charles-François Natali added the comment:
>
> > However, the reason I'm keen on iterdir_stat() is that I'm seeing it
> speed up os.walk() by a factor of 10 in my recent tests (note that I've
> made local mods, so these results aren't reproducible for others yet). This
> is doing a walk on a dir tree with 7800 files and 155 dirs:
> >
> > Using fast _betterwalk
> > Priming the system's cache...
> > Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 1/3...
> > Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 2/3...
> > Benchmarking walks on C:\Work\betterwalk\benchtree, repeat 3/3...
> > os.walk took 0.178s, BetterWalk took 0.017s -- 10.5x as fast
> >
> > Sometimes Windows will go into this "I'm really caching stat results
> good" mode -- I don't know what heuristic determines this -- and then I'm
> seeing a 40x speed increase. And no, you didn't read that wrong. :-)
>
> I/O benchmarks shouldn't use timeit or repeated calls: after the first
> run, most of your data is in cache, so subsequent runs are
> meaningless.
>
> I don't know about Windows, but on Linux you should do something like:
> # echo 3 > /proc/sys/vm/drop_caches
>
> to start out clean.
>
> ----------
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue11406>
> _______________________________________
>

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11406>
_______________________________________