[Python-ideas] BetterWalk, a better and faster os.walk() for Python

Andrew Barnert abarnert at yahoo.com
Fri Nov 23 01:05:39 CET 2012

From: Ben Hoyt <benhoyt at gmail.com>
Sent: Thu, November 22, 2012 2:44:02 PM

> > I tested on OS X 10.8.2 on a Retina MBP 15" with 16GB and the stock SSD,  
> > Apple 2.6 and 2.7 and python.org 3.3. It seems to be a bit slower in 2.x,  a 
> > faster in 3.x, more so in 32-bit mode, and better without -s. The  best 
>result I
> > got anywhere was 1.5x (3.3, 32-bit, no -s), but repeating  that test gave
> > anywhere from 1.2x to 1.5x.
> Yeah, that's about  what I'm seeing on Linux and OS X. (Though for some
> weird reason I'm seeing  10x as fast on OS X when I do "python
> benchmark.py /usr" -- hence my comments in the  README.)

I get exactly 1.0x on this test with 2.6 and 2.7, 1.3x with 3.3, 1.4x with 
32-bit 3.3. Any chance your /usr has a symlink to a remote or otherwise slow or 
non-HFS+ filesystem? Is that worth testing?

Also, the -s version seems to fail on dangling symlinks:

$ python2.7 benchmark.py -s /usr
Priming the system's cache...Benchmarking walks on /usr, repeat 1/3...
Traceback (most recent call last):
  File "benchmark.py", line 121, in <module>
  File "benchmark.py", line 118, in main
    benchmark(tree_dir, get_size=options.size)
  File "benchmark.py", line 83, in benchmark
    os_walk_time = min(os_walk_time, timeit.timeit(do_os_walk, number=1))
 line 228, in timeit
    return Timer(stmt, setup, timer).timeit(number)
 line 194, in timeit
    timing = self.inner(it, self.timer)
 line 100, in inner
  File "benchmark.py", line 57, in do_os_walk
    size += os.path.getsize(fullname)
 line 49, in getsize
    return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: 

$ readlink 

$ ls 

 No such file or directory

More information about the Python-ideas mailing list