[Python-ideas] Please consider skipping hidden directories in os.walk, os.fwalk, etc.

[Yuval Greenfield]

On Mon, May 7, 2018 at 9:44 PM Steve Barnes <gadgetsteve at live.co.uk> wrote:

> Since the implementation of os.walk has changed to use os.scandir which
> exposes the returned file statuses in the os.DirEntry.stat() the
> overhead should be minimal.
>
> An alternative would be to add another new function, say os.vwalk(), to
> only walk visible entries.
>

On Tue, May 8, 2018 at 12:06 AM Steven D'Aprano <steve at pearwood.info> wrote:

> I would write something like:
>     for root, dirs, files in filter(ignorable, os.walk(some_dir)):


I agree with Steven with regards to `filter` needing to be flexible. If you
want to avoid duplicate `stat` calls, you'll probably write:

import os
import stat
def is_hidden(st):
    return bool(st.st_file_attributes & stat.FILE_ATTRIBUTE_HIDDEN)
def visible_walk(path):
    for entry in os.scandir(path):
        if entry.is_dir():
            if not is_hidden(entry.stat()):
                yield from visible_walk(entry.path)
        else:
            if not is_hidden(entry.stat()):
                yield entry.path


Then you can decide whether you want to ignore hidden files or just hidden
directories. The variations for such a need are many. So it makes sense to
leave any specific filtering need outside of the standard library. A PyPI
package with a few standard filtered walks could be a nice exploration for
this idea.

Cheers,

Yuval
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20180508/ca481db0/attachment.html>