[Python-Dev] PEP 471: scandir(fd) and pathlib.Path(name, dir_fd=None)

[Ben Hoyt]

Thanks, Victor.

I don't have any experience with dir_fd handling, so unfortunately
can't really comment here.

What advantages does it bring? I notice that even os.listdir() on
Python 3.4 doesn't have anything related to file descriptors, so I'd
be in favour of not including support. We can always add it later.

-Ben

On Tue, Jul 1, 2014 at 3:44 AM, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
>
> IMO we must decide if scandir() must support or not file descriptor.
> It's an important decision which has an important impact on the API.
>
>
> To support scandir(fd), the minimum is to store dir_fd in DirEntry:
> dir_fd would be None for scandir(str).
>
>
> scandir(fd) must not close the file descriptor, it should be done by
> the caller. Handling the lifetime of the file descriptor is a
> difficult problem, it's better to let the user decide how to handle
> it.
>
> There is the problem of the limit of open file descriptors, usually
> 1024 but it can be lower. It *can* be an issue for very deep file
> hierarchy.
>
> If we choose to support scandir(fd), it's probably safer to not use
> scandir(fd) by default in os.walk() (use scandir(str) instead), wait
> until the feature is well tested, corner cases are well known, etc.
>
>
> The second step is to enhance pathlib.Path to support an optional file
> descriptor. Path already has methods on filenames like chmod(),
> exists(), rename(), etc.
>
>
> Example:
>
> fd = os.open(path, os.O_DIRECTORY)
> try:
>    for entry in os.scandir(fd):
>       # ... use entry to benefit of entry cache: is_dir(), lstat_result ...
>       path = pathlib.Path(entry.name, dir_fd=entry.dir_fd)
>       # ... use path which uses dir_fd ...
> finally:
>     os.close(fd)
>
> Problem: if the path object is stored somewhere and use after the
> loop, Path methods will fail because dir_fd was closed. It's even
> worse if a new directory uses the same file descriptor :-/ (security
> issue, or at least tricky bugs!)
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com