[Python-Dev] os.walk() is going to be *fast* with scandir

[Ben Hoyt]

Just thought I'd share some of my excitement about how fast the all-C
version [1] of os.scandir() is turning out to be.

Below are the results of my scandir / walk benchmark run with three
different versions. I'm using an SSD, which seems to make it
especially faster than listdir / walk. Note that benchmark results can
vary a lot, depending on operating system, file system, hard drive
type, and the OS's caching state.

Anyway, os.walk() can be FIFTY times as fast using os.scandir().

# Old ctypes implementation of scandir in scandir.py:
C:\work\scandir>\work\python\cpython\python benchmark.py -r
Using slower ctypes version of scandir
os.walk took 1.144s, scandir.walk took 0.060s -- 19.2x as fast

# Existing "half C" implementation of scandir in _scandir.c:
C:\work\scandir>\Python34-x86\python.exe benchmark.py -r
Using fast C version of scandir
os.walk took 1.160s, scandir.walk took 0.042s -- 27.6x as fast

# New "all C" os.scandir implementation in posixmodule.c:
C:\work\scandir>\work\python\cpython\python benchmark.py -r
Using Python 3.5's builtin os.scandir()
os.walk took 1.141s, scandir.walk took 0.022s -- 53.0x as fast

[1] Work in progress implementation as part of Python 3.5's
posixmodule.c available here:
https://github.com/benhoyt/scandir/blob/master/posixmodule.c

-Ben