[Python-ideas] BetterWalk, a better and faster os.walk() for Python

[Andrew Barnert]

From: Ben Hoyt <benhoyt at gmail.com>
Sent: Thu, November 22, 2012 3:40:00 AM

> In any case, I'd love it if folks could run the benchmark on  their
> system (with and without -s) and comment further on the idea and  API.


I tested on OS X 10.8.2 on a Retina MBP 15" with 16GB and the stock SSD, using 
Apple 2.6 and 2.7 and python.org 3.3. It seems to be a bit slower in 2.x, a bit 
faster in 3.x, more so in 32-bit mode, and better without -s. The best result I 
got anywhere was 1.5x (3.3, 32-bit, no -s), but repeating that test gave 
anywhere from 1.2x to 1.5x.

Here are the last test runs for each run:

Retina:betterwalk abarnert$ python2.6 benchmark.py
Priming the system's cache...
Benchmarking walks on benchtree, repeat 1/3...
Benchmarking walks on benchtree, repeat 2/3...
Benchmarking walks on benchtree, repeat 3/3...
os.walk took 0.057s, BetterWalk took 0.061s -- 0.9x as fast

Retina:betterwalk abarnert$ python2.7 benchmark.py 
Creating tree at benchtree: depth=4, num_dirs=5, num_files=50
Priming the system's cache...
Benchmarking walks on benchtree, repeat 1/3...
Benchmarking walks on benchtree, repeat 2/3...
Benchmarking walks on benchtree, repeat 3/3...
os.walk took 0.059s, BetterWalk took 0.066s -- 0.9x as fast

Retina:betterwalk abarnert$ python3.3 benchmark.py 
Priming the system's cache...
Benchmarking walks on benchtree, repeat 1/3...
Benchmarking walks on benchtree, repeat 2/3...
Benchmarking walks on benchtree, repeat 3/3...
os.walk took 0.074s, BetterWalk took 0.058s -- 1.3x as fast

Retina:betterwalk abarnert$ python2.6 benchmark.py -s
Priming the system's cache...
Benchmarking walks on benchtree, repeat 1/3...
Benchmarking walks on benchtree, repeat 2/3...
Benchmarking walks on benchtree, repeat 3/3...
os.walk size 226395000, BetterWalk size 226395000 -- equal
os.walk took 0.097s, BetterWalk took 0.104s -- 0.9x as fast

Retina:betterwalk abarnert$ python2.7 benchmark.py -s
Priming the system's cache...
Benchmarking walks on benchtree, repeat 1/3...
Benchmarking walks on benchtree, repeat 2/3...
Benchmarking walks on benchtree, repeat 3/3...
os.walk size 226395000, BetterWalk size 226395000 -- equal
os.walk took 0.100s, BetterWalk took 0.109s -- 0.9x as fast

Retina:betterwalk abarnert$ python3.3 benchmark.py -s
Priming the system's cache...
Benchmarking walks on benchtree, repeat 1/3...
Benchmarking walks on benchtree, repeat 2/3...
Benchmarking walks on benchtree, repeat 3/3...
os.walk size 226395000, BetterWalk size 226395000 -- equal
os.walk took 0.121s, BetterWalk took 0.099s -- 1.2x as fast

Retina:betterwalk abarnert$ python3.3-32 benchmark.py
Priming the system's cache...
Benchmarking walks on benchtree, repeat 1/3...
Benchmarking walks on benchtree, repeat 2/3...
Benchmarking walks on benchtree, repeat 3/3...
os.walk took 0.073s, BetterWalk took 0.048s -- 1.5x as fast

Retina:betterwalk abarnert$ python3.3-32 benchmark.py -s
Priming the system's cache...
Benchmarking walks on benchtree, repeat 1/3...
Benchmarking walks on benchtree, repeat 2/3...
Benchmarking walks on benchtree, repeat 3/3...
os.walk size 226395000, BetterWalk size 226395000 -- equal
os.walk took 0.129s, BetterWalk took 0.100s -- 1.3x as fast