On 5/3/2017 7:13 PM, Koos Zevenhoven wrote:

On Thu, May 4, 2017 at 1:07 AM, Terry Reedy <tjreedy@udel.edu> wrote:

...

On 5/3/2017 2:15 PM, Brett Cannon wrote:

...

My allergies have hit me hard so I'm not thinking at full capacity, but did we ever decide if supporting os.PathLike in the stdlib was viewed as an enhancement or bugfix? Specifically I'm thinking of https://bugs.python.org/issue30218 for adding support to shutil.unpack_archive() and whether it should be backported to 3.6.

On the face of it, that particular issue looks like an enhancement that should have gone into 3.6

Agreed.

...

(if ever), but did not. I notice that https://www.python.org/dev/peps/pep-0519/#implementation did not include "Update shutil", so it was not done, at least not completely.

Shutil was among the most important to be updated, IMO.

I had made some sort of list of affected modules elsewhere [1]: ntpath, posixpath, os.scandir, os.[other stuff], DirEntry (tempted to say os.DirEntry, but that is not true), shutil.[stuff], (io.)open, fileinput, filecmp, zipfile, tarfile, tempfile (for the 'dir' keyword arguments), maybe even glob and fnmatch (are the patterns paths?)

What did not get done for 3.6 should be proposed for 3.7.

It looks like what made it to PEP519 was mainly this:

"It is expected that most APIs in Python's standard library that currently accept a file system path will be updated appropriately to accept path objects (whether that requires code or simply an update to documentation will vary)."

To me that says that not being updated (yet) is not a bug.

...

Was shutil updated at all? Is unpack_archive the only shutil function not updated? If so, I could see the omission as a bug.

If the patch for 30218 were applied in 3.6, would the doc https://docs.python.org/3/library/shutil.html#shutil.unpack_archive need to be changed, with a note "Added in 3.6.2: filename can be any pathlike object"? If so, it is an enhancement.

Regardless of bugfix vs enhancement semantics, that seems like a good thing to do.

Enhancing public APIs in normal (non-provisional) modules in bugfix releases has turned out to be a bad thing to do. Hence the policy to not do that. The few exceptions have been necessary to fix a bug that needed to be fixed, and could not reasonably be fixed otherwise.

[1] e.g. in this thread somewhere: https://mail.python.org/pipermail/python-ideas/2016-April/039827.html

-- Terry Jan Reedy

On 5/4/2017 10:43 AM, Koos Zevenhoven wrote:

On Thu, May 4, 2017 at 4:19 AM, Terry Reedy <tjreedy@udel.edu> wrote:

...

What did not get done for 3.6 should be proposed for 3.7.

Anyone, feel free. The nightmare part is done, so this could be a case where a PR actually pays off in terms of being able to use the feature. There's no need for any unnecessary masochism (should there ever be?).

I have no idea what this means, but don't need to.

...

Enhancing public APIs in normal (non-provisional) modules in bugfix releases has turned out to be a bad thing to do. Hence the policy to not do that. The few exceptions have been necessary to fix a bug that needed to be fixed, and could not reasonably be fixed otherwise.

Such exceptions can of course more easily be made when the adoption of a version is still small, and almost all users will never see X.Y.0 or X.Y.1.

This is not an allowed excuse for breaking the policy. The x.y language is defined when x.y.0 is released. Please stop. -- Terry Jan Reedy

Thanks for the feedback, everyone. I'll just consider it an enhancement then. On Wed, 3 May 2017 at 15:07 Terry Reedy <tjreedy@udel.edu> wrote:

On 5/3/2017 2:15 PM, Brett Cannon wrote:

...

My allergies have hit me hard so I'm not thinking at full capacity, but did we ever decide if supporting os.PathLike in the stdlib was viewed as an enhancement or bugfix? Specifically I'm thinking of https://bugs.python.org/issue30218 for adding support to shutil.unpack_archive() and whether it should be backported to 3.6.

On the face of it, that particular issue looks like an enhancement that should have gone into 3.6 (if ever), but did not. I notice that https://www.python.org/dev/peps/pep-0519/#implementation did not include "Update shutil", so it was not done, at least not completely.

Was shutil updated at all?

Probably not explicitly. A lot of support just fell through thanks to updating os and os.path to support path-like objects.

Is unpack_archive the only shutil function not updated?

Not sure. I opened https://bugs.python.org/issue30235 to track finding out. -Brett If so, I could see the omission as a bug.

If the patch for 30218 were applied in 3.6, would the doc https://docs.python.org/3/library/shutil.html#shutil.unpack_archive need to be changed, with a note "Added in 3.6.2: filename can be any pathlike object"? If so, it is an enhancement.

-- Terry Jan Reedy

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

On 4 May 2017 at 08:25, Victor Stinner <victor.stinner@gmail.com> wrote:

If you start to backport support for the fspath protocol, be prepared to have to backport it *many* places. I expect that slowly in the near future, many functions will be patched to support the fspath protocol.

I suggest to only do that in master. It's not that hard to cast manually to a string: it's just str(path), no?

For 3.6+, "os.fspath(path)" would be more robust - unlike str(), it will still error out immediately if you pass it something completely inappropriate. That improved casting mechanism and the implicit support in the low level APIs is the main benefit I see in PEP 519, and if we were talking about an os module API that was missing os.path support, I'd be more likely to be on the "it's a bug" side of things. It's only higher level APIs like shutil that I put into the same category as third party libraries for now: Python 3.6 users shouldn't expect implicit use of the fspath protocol to be universal yet, so they may still need some explicit calls to os.fspath() in their own code to get different APIs to work well together. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 5/5/2017 3:36 PM, Chris Barker wrote:

Sorry to come late to the game, It wasn't immediately clear to me what the implications were of the "enhancement or bugfix" distinction...

On Thu, May 4, 2017 at 9:46 PM, Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote:

That improved casting mechanism and the implicit support in the low level APIs is the main benefit I see in PEP 519, and if we were talking about an os module API that was missing os.path support, I'd be more likely to be on the "it's a bug" side of things.

absolutely.

It's only higher level APIs like shutil that I put into the same category as third party libraries for now: Python 3.6 users shouldn't expect implicit use of the fspath protocol to be universal yet,

I think stdlib packages, like shutil are very much distinct from third party libs, and users certainly expect the entire stdlib to be updated to support new language features.

We very often make the distinction between third party libs and stdlibs -- in fact, that is one reason we are reluctant to add new packages to the stdlib...

Indeed, IIRC, that was the entire motivation for PEP 519 -- it started as a "let's make Path objects work with the stdlib", but when we looked at how to do that, it became clear that new protocol was teh best way to do that and also provide flexibility to do that.

And the PEP states:

""" Changes to Python's standard library are also proposed to utilize this protocol where appropriate to facilitate the use of path objects where historically only str and/or bytes file system paths are accepted. The goal is to facilitate the migration of users towards rich path objects while providing an easy way to work with code expecting str or bytes . """

It doesn't actually say "everywhere possible in the stdlib", but if the goal is to facilitate migration, as stated, then the any but truly obscure functions should be covered -- and shutil is certainly not obscure.

So it would be really great if any updates to shutils (and other stdlib packages) to support the new protocol be back-ported.

Yes, it's "just" adding an extra call, but in my experience, low barrier to entry are enough to discourage adoption -- and handful of shutil function failing will certainly be enough to keep some folks from adopting the new approach.

Is there any chance of it breaking already working code? That would be the one compelling reason to not back-port.

The problem is not breaking existing code. The problem is that when someone writes code a year from now and tests it under Python 3.6.2, then a customer of theirs finds it doesn't work in 3.6.1. This will happen if 3.6.2 supports Path parameters to functions that 3.6.1 does not. We've burned ourselves before with this, most famously with True and False in 2.2.1: http://python-history.blogspot.com/2013/11/the-history-of-bool-true-and-fals... -- Eric.

On Fri, May 5, 2017 at 2:34 PM, Terry Reedy <tjreedy@udel.edu> wrote:

So it would be really great if any updates to shutils (and other stdlib

...

packages) to support the new protocol be back-ported.

Not if they change the language.

we're not talking about language changes here -- we are talking about updates to the stdlib. Eric wrote: The problem is that when someone writes code a year from now and tests it

under Python 3.6.2, then a customer of theirs finds it doesn't work in 3.6.1. This will happen if 3.6.2 supports Path parameters to functions that 3.6.1 does not.

This does, of course apply to the stdlib -- not just the language. We've burned ourselves before with this, most famously with True and False

in 2.2.1: http://python-history.blogspot.com/2013/11/the-history-of- bool-true-and-false.html

I'd argue that adding three built-ins is a much bigger deal than enhancing a subset of one standard package -- not that the same issues wouldn't arise. I think it's still worth, but I can certainly see why it might not be. Too bad we didn't do a bit more testing for completely when it was beta.... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 6 May 2017 at 05:36, Chris Barker <chris.barker@noaa.gov> wrote:

It doesn't actually say "everywhere possible in the stdlib", but if the goal is to facilitate migration, as stated, then the any but truly obscure functions should be covered -- and shutil is certainly not obscure.

So it would be really great if any updates to shutils (and other stdlib packages) to support the new protocol be back-ported.

This argument would apply to all the other protocols we've added over time, and would lead to significant language churn in maintenance releases if we regularly relied on it. As another example from 3.6, we found that the non-type metaclasses in the standard library (most notably abc.ABCMeta) don't play nice with the arbitrary keyword support in __init_subclass__, since they don't pass the additional keywords from the class definition up to type.__new__ (which in turn would pass them along to __init_subclass__). That's an oversight we're definitely going to fix, but we'll fix it in 3.7, not in 3.6.x.

Yes, it's "just" adding an extra call, but in my experience, low barrier to entry are enough to discourage adoption -- and handful of shutil function failing will certainly be enough to keep some folks from adopting the new approach.

The critical problem prior to 3.6 wasn't the extra call, it was that the recommended extra call had the undesirable side effect of allowing *any* object to be used as a "file system path". As a result, you couldn't easily separate the check out into a helper function without putting a lot of annoying "isinstance" checks into it (or else relying on functools.singledispatch). By contrast, 3.6 users can just unconditionally call os.fspath(), and know the result will work with all stdlib APIs, without allowing nonsense like attempting to use "str(str)" as a filesystem path. Doing that implicitly in various APIs is certainly a nice UX enhancement, but it's the replacement of str with os.fspath as the recommended coercion function that removes the major barrier to pathlib adoption.

Is there any chance of it breaking already working code? That would be the one compelling reason to not back-port.

"Don't make the third party compatibility testing matrix impossibly complicated" is our other major reason not to backport new features. We already have platform providers supporting 3.6.0 in the wild (Heroku and AWS Lambda), and we know 3.6.1 is going to be in at least Fedora 26, and probably Ubuntu 17.10. We want CI providers like Travis/GitLab/Circle CI/etc to be able to provide a *single* 3.6.x release (typically the most recent one), and have the guarantee be "code that passes on this maintenance release will only fail on earlier maintenance releases if you hit a genuine bug in those releases". Changing a function signature from accepting "Union[str, bytes]" (or potentially even just "str") to instead accepting "os.PathLike" doesn't count as fixing a genuine bug in the earlier releases - it just means we made the API more permissive in a maintenance release. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 9 May 2017 at 05:08, Chris Barker <chris.barker@noaa.gov> wrote:

On Fri, May 5, 2017 at 9:30 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

By contrast, 3.6 users can just unconditionally call os.fspath(), and know the result will work with all stdlib APIs, without allowing nonsense like attempting to use "str(str)" as a filesystem path.

Doing that implicitly in various APIs is certainly a nice UX enhancement, but it's the replacement of str with os.fspath as the recommended coercion function that removes the major barrier to pathlib adoption.

honestly, I'm not sure about that -- burying a str() call in a lib is clearly risky, but using one in your user code, not so much -- you usually know you have a path-like object (if not a Path) itself. the extra need for wrapping str() around things was a barrier to pathlib adoption -- and it doesn't take much of a barrier to reduce the adoption of something new.

Exactly - the key barrier to adoption was the lack of "os.fspath()", since it meant that neither the standard library *nor* third party helper libraries could reliably deal with the conversion task on behalf of their users. Instead, people had to carefully manage their own code, ensuring both that they used str() in the appropriate places, *and* that they didn't accidentally call str() on the wrong thing. os.fspath() changes that to be much simpler and friendlier to an iterative development process: 1. By default, expect things to just work 2. If they blow up with TypeError, wrap the offending argument in a call to os.fspath() 3. (optional) file an RFE with the maintainers of the affected API to call os.fspath() implicitly on that argument The number of APIs that fall into the second category is going to be non-zero for quite a while (both in the standard library and elsewhere), but enough folks proceeding to step 3 will bring it down over time. Outside customers complaining to their commercial support vendors though, it's rarely helpful to insist to maintainers (whether of standard library modules or third party ones) that their failure to adopt a feature only added in the latest version of Python now counts as a bug in their software. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 04.05.17 21:01, Berker Peksağ wrote:

We've already backported a few patches that improves the PEP 519 support in the stdlib with the permission from the release manager of 3.6. I'd ask Ned whether bpo-30218 qualifies for backporting to 3.6.

AFAIK it was before releasing 3.6.1. Some users avoid using x.0 versions of software. For 3.6.2 we can have stronger rule.