[Python-ideas] BetterWalk, a better and faster os.walk() for Python

[Mike Meyer]

On Thu, Nov 22, 2012 at 5:39 AM, Ben Hoyt <benhoyt at gmail.com> wrote:
> In the recent thread I started called "Speed up os.walk()..." [1] I
> was encouraged to create a module to flesh out the idea, so I present
> you with BetterWalk:
>
> https://github.com/benhoyt/betterwalk#readme
>
> It's basically all there, and works on Windows, Linux, and Mac OS X.
> It probably works on FreeBSD too, but I haven't tested that.

It doesn't work on FreeBSD 9.1. Here's a quick failure in  the source
directory that might help
you diagnose the problem (I probably won't be able to look into it
deeper until after the holiday weekend):

>>>from betterwalk import walk
>>>for x in walk('.'):
... print x
...
('.', [u'', u'', u'erwalk.py', u'erwalk.pyc', u'p.py'], [u'ignore',
u'hmark.py', u's', u'erwalk.py', u'htree', u'attributes', u'GES.txt',
u'ME.md', u'NSE.txt']) (u'./', [u'', u'', u'erwalk.py', u'erwalk.pyc',
u'p.py'], [u'ignore', u'hmark.py', u's', u'erwalk.py', u'htree',
u'attributes', u'GES.txt', u'ME.md', u'NSE.txt'])
<break>

I also got an error trying to run the benchmark program on 3.2 (this
is python-ideas, which means things discussed here are bound for
Python 3):

bhuda% python3.2 benchmark.py
Creating tree at benchtree: depth=4, num_dirs=5, num_files=50
Traceback (most recent call last):
  File "benchmark.py", line 121, in <module>
    main()
  File "benchmark.py", line 116, in main
    create_tree(tree_dir)
  File "benchmark.py", line 26, in create_tree
    f.write(line * 20000)
TypeError: 'str' does not support the buffer interface

You need to use a bytestring instead of a string here. The best way
will depend on how old a version of python you want to support.

     <mike