[Python-Dev] Re: Request for reviewers: pathlib improvements

Hi everyone, It’s been a fairly quiet couple of months in pathlib work from myself - just a couple of forum posts. Kevin Follstad has been beavering away on a PR https://github.com/python/cpython/pull/26906 to address bpo-24132 https://bugs.python.org/issue24132, I’ve promised him a review, though I’m already late! I’d like to request core review of 2 PRs, having already bumped the bugs / forum threads and waited the statuatory 5 weeks. They are: - PR 26153 https://github.com/python/cpython/pull/26153 - document and test pathlib.Path.absolute() - PR 25701 https://github.com/python/cpython/pull/25701 - remove pathlib._Accessor The first introduces documentation and tests for a feature that has long existed and is in widespread use. The second removes an internal indirection layer that complicates the code (and therefore work on other pathlib tickets). Very appreciate to any core dev who might be able to find the time for these. Thank you! Barney On Sat, 24 Apr 2021 at 23:16, Barney Gale wrote:

Hi all,

I hope I’m not in danger of abusing this mailing list - please let me know if this is too much noise.

I’m very happy to share that 3 more pathlib PRs have landed - #18909 https://github.com/python/cpython/pull/18909, #25240 https://github.com/python/cpython/pull/25240 and #25277 https://github.com/python/cpython/pull/25277. Thanks for the reviews - I appreciate reviewer time is scarce!

I may be pushing my luck here, but there are *just two* PRs outstanding before work on AbstractPath can begin in earnest.

I would love for these to land before the 3.10 feature freeze so that we can spend the entire 3.11 development cycle getting the design/implementation of AbstractPath exactly right. The PRs are:

- #25271 https://github.com/python/cpython/pull/25271 / bpo-42998 https://bugs.python.org/issue42998 - Add ‘user’ parameter to Path.home() - A simple patch. No reviewers yet - #25264 https://github.com/python/cpython/pull/25264 / bpo-43757 https://bugs.python.org/issue43757 - Make Path.resolve() use os.path.realpath() to resolve all symlinks in a path - A more complex patch with several contributions from reviewers - Adds keyword-only ‘strict’ param to realpath() - Now backwards compatible! No longer changes default realpath() behaviour

If anyone can spare the time over the next week to help get these patches over the line, I would really appreciate it!

A quick elevator pitch for why AbstractPath is an important future feature:

- pathlib is a wonderful library that beautifully shows off Python’s “everything is an object” approach. It’s rightly front-and-centre of many Python beginner tutorials. If we can extend that “everything” to include files in tarballs, zip files, disc images, FTP servers, S3, etc, it allows Python users to easily apply their pathlib knowledge to new (and common!) tasks. - In DevOps it’s common to deal with remote or embedded filesystems. There’s a need for pathlib-like interfaces, and that need is already met in popular third-party packages either by *emulating* Path APIs (often with subtle differences) or *hackily extending* Path/_Accessor/_Flavour (which is brittle and totally unsupported). Even in the standard library, zipfile.Path is a mostly-compatible emulation of pathlib.Path. Providing a supported route for extending an AbstractPath saves a bunch of heartache and makes the wider pathlib ecosystem more robust.

Thanks again to everyone who has contributed their time and expertise up to this point.

Cheers

Barney

On Fri, 9 Apr 2021 at 01:17, Barney Gale wrote:

...

Thanks so much to everyone who discussed and reviewed the code and made suggestions.

The bulk of these patches have now landed. For those following along at home, here’s a summary of the remaining PRs/bugs :

- #18909 https://github.com/python/cpython/pull/18909 / bpo-39950 https://bugs.python.org/issue39950: Add Path.hardlink_to() method that supersedes link_to() - Note: already under core review - #25271 https://github.com/python/cpython/pull/25271 / bpo-42998 https://bugs.python.org/issue42998: Add ‘user’ parameter to Path.home() - #25240 https://github.com/python/cpython/pull/25240 / bpo-40107 https://bugs.python.org/issue40107: Stop using os.open() to implement Path.open() - #25264 https://github.com/python/cpython/pull/25264 / bpo-43757 https://bugs.python.org/issue43757: Make Path.resolve() use os.path.realpath() to resolve all symlinks in a path - Note: changes realpath() behaviour when symlink loops are encountered (!), and adds a keyword-only 'strict' param. - #25277 https://github.com/python/cpython/pull/25277 / bpo-39899 https://bugs.python.org/issue39899: Follow-up fix for oversight in #18841 https://github.com/python/cpython/pull/18841

I believe this is a complete account of the outstanding prep work. Once these land, pathlib’s *path*, *flavour* and *accessor* abstractions will be less leaky, which allows work on pathlib.AbstractPath to proceed.

Thanks again to Antoine Pitrou, Brett Cannon, Serhiy Storchaka, Steve Dower and Eryk Sun for their invaluable contributions.

Barney

On Sat, 3 Apr 2021 at 03:02, Barney Gale wrote:

...

I’m working towards supporting subclassing pathlib.Path for things like zip/tar archives, iso files, S3, etc. This was first brought up a few years ago in bpo-24132 https://bugs.python.org/issue24132 and discussed further on the python-ideas forum https://discuss.python.org/t/make-pathlib-extensible/3428.

Before I can approach the meat of the issue, I'd like to land a handful of backwards-compatible patches that tidy up the pathlib internals. These are:

- #19220 https://github.com/python/cpython/pull/19220 / bpo-39924 https://bugs.python.org/issue39924: Handle missing os functions more consistently in pathlib - #18838 https://github.com/python/cpython/pull/18838 / bpo-39895 https://bugs.python.org/issue39895: Move pathlib.Path.touch() implementation into the path accessor - #19342 https://github.com/python/cpython/pull/19342 / bpo-40038 https://bugs.python.org/issue40038: Remove partial support for preserving accessor when modifying a path - #18841 https://github.com/python/cpython/pull/18841 / bpo-39899 https://bugs.python.org/issue39899: Make pathlib use os.path.expanduser() to expand home directories - #18834 https://github.com/python/cpython/pull/18834 / bpo-39659 https://bugs.python.org/issue39659: Route calls from pathlib.Path to os.getcwd() via the path accessor - #18864 https://github.com/python/cpython/pull/18864 / bpo-39906 https://bugs.python.org/issue39906: Add follow_symlinks parameter to pathlib.Path.stat() and chmod() - #24294 https://github.com/python/cpython/pull/24294 / bpo-42999 https://bugs.python.org/issue42999: Expand and clarify pathlib.Path.link_to() documentation

Could I please request review of these patches? A couple may already be ready to land.

(I appreciate they’re a little dry. This is all prep work for the introduction of a shiny new class, tentatively named ‘AbstractPath’).

Thanks!

Barney