[Python-ideas] BetterWalk, a better and faster os.walk() for Python

[Stefan Drees]

Hi Ben,

On 22.11.12 21:23, Ben Hoyt wrote:
>> On Mageia Linux 1, I had the following results:
>> - Python 2.7: 0.7x as fast
>> - Python 3.2: 1.1x as fast
>> - Python 3.3: 1.2x as fast
>> The -s flag didn't make a difference.
>
> Thanks. Out of interest, 64 bit or 32 bit (system and Python)?
>
> I wonder if ctypes got significantly faster in Python 3.x, or what's
> going on here. Python 3 is significantly faster in my tests too --
> noticeable on Linux.
>
>> Do note that the benchmark is very fast - around ~40 ms for a walk.
>
> Yeah, that's over too quickly for a real test, isn't it? I'm already
> creating a 230MB dir, but maybe I need to add more, smaller files.
> Easy to tweak with the NUM_FILES and NUM_DIRS constants (though you'll
> need to delete your current "benchtree" dir so it recreates it. Maybe
> I should bump up the defaults too.
>

thanks for providing a fix. Now using revision  f975b2a5... with fixed 
tree creation on a Mac BookPro (8 GB RAM) and OS X 10.8.2 no real-time 
virus scanning during testrun ;-) walking a solid-state disk:

Oh, and I varied a bit the constants ...

With Python 3.3.0:

  + depth=4, num_dirs=5, num_files=50
    os.walk took 0.122s, BetterWalk took 0.076s -- 1.6x as fast

  + depth=4, num_dirs=5, num_files=100
    os.walk took 0.142s, BetterWalk took 0.098s -- 1.4x as fast

  + depth=4, num_dirs=10, num_files=50
    os.walk took 0.840s, BetterWalk took 0.634s -- 1.3x as fast

  + depth=5, num_dirs=5, num_files=50
    os.walk took 0.617s, BetterWalk took 0.446s -- 1.4x as fast

With Python 2.7.3:

  + depth=4, num_dirs=5, num_files=50
    os.walk took 0.060s, BetterWalk took 0.059s -- 1.0x as fast

  + depth=4, num_dirs=5, num_files=100
    os.walk took 0.121s, BetterWalk took 0.136s -- 0.9x as fast

  + depth=4, num_dirs=10, num_files=50
    os.walk took 0.658s, BetterWalk took 0.664s -- 1.0x as fast

  + depth=5, num_dirs=5, num_files=50
    os.walk took 0.473s, BetterWalk took 0.506s -- 0.9x as fast

All the best,
Stefan.