[Python-ideas] BetterWalk, a better and faster os.walk() for Python

[Mike Meyer]

On Thu, Nov 22, 2012 at 1:23 PM, Ben Hoyt <benhoyt at gmail.com> wrote:
>> It doesn't work on FreeBSD 9.1. Here's a quick failure in  the source
> ...
>> ('.', [u'', u'', u'erwalk.py', u'erwalk.pyc', u'p.py'], [u'ignore',
>
> Thanks. FreeBSD seemed to have the same "problem" as Mac OS X (issue
> #2). I fixed this on the GitHub repo now -- please try again.

That's not surprising. OSX started life as Mach with the BSD
personality and a FreeBSD userland.

>> I also got [a TypeError] trying to run the benchmark program on 3.2 (this
>> is python-ideas, which means things discussed here are bound for
>> Python 3):
> Yeah, sorry about that. I'd certainly tested the benchmark on Python
> 3.x, however not the directory tree creation. Again, fixed now on
> GitHub.

Both things seem to be fixed. On a zfs file system, I get

python3.2:  	 1.8x as fast
python2.7:	 1.3x as fast

There are two that worry me, though:

python3.2:    	   0.8x as fast
python 2.7:	   0.6x as fast

I get the same results on an nfs mount of a zfs file system (the
remote fs should not matter) and an memory backed file system
(typically used for /tmp). I had hunt for a disk-based fs to get the
first set of results :-(.

I suspect that neither of these have d_type on the fs, so we're seeing
a serious performance hit for systems that don't have d_type. That
certainly bears further investigation. Could it just be the
python/ctype implementation vs. native code?

      <mike