[Python-ideas] BetterWalk, a better and faster os.walk() for Python
mwm at mired.org
Thu Nov 22 21:36:22 CET 2012
On Thu, Nov 22, 2012 at 1:23 PM, Ben Hoyt <benhoyt at gmail.com> wrote:
>> It doesn't work on FreeBSD 9.1. Here's a quick failure in the source
>> ('.', [u'', u'', u'erwalk.py', u'erwalk.pyc', u'p.py'], [u'ignore',
> Thanks. FreeBSD seemed to have the same "problem" as Mac OS X (issue
> #2). I fixed this on the GitHub repo now -- please try again.
That's not surprising. OSX started life as Mach with the BSD
personality and a FreeBSD userland.
>> I also got [a TypeError] trying to run the benchmark program on 3.2 (this
>> is python-ideas, which means things discussed here are bound for
>> Python 3):
> Yeah, sorry about that. I'd certainly tested the benchmark on Python
> 3.x, however not the directory tree creation. Again, fixed now on
Both things seem to be fixed. On a zfs file system, I get
python3.2: 1.8x as fast
python2.7: 1.3x as fast
There are two that worry me, though:
python3.2: 0.8x as fast
python 2.7: 0.6x as fast
I get the same results on an nfs mount of a zfs file system (the
remote fs should not matter) and an memory backed file system
(typically used for /tmp). I had hunt for a disk-based fs to get the
first set of results :-(.
I suspect that neither of these have d_type on the fs, so we're seeing
a serious performance hit for systems that don't have d_type. That
certainly bears further investigation. Could it just be the
python/ctype implementation vs. native code?
More information about the Python-ideas