[Distutils]PEP 518 - pyproject.toml with no build-system.requires
Hey everyone! In PEP 518, it is not clearly specified how a project that has a pyproject.toml file but has no build-system.requires should be treated (i.e. build-system table). In pip 10, such a pyproject.toml file was allowed and built with setuptools and wheel, which has resulted in a lot of projects making releases that assumed that such a pyproject.toml file is valid and they use setuptools and wheel. I understand that at least pytest, towncrier and Twisted might have done so. This happened since these projects have included configuration for some tool in pyproject.toml (some of which use only pyproject.toml for configuration -- black, towncrier). There's a little bit of subtlety here, in pip 10's implementation: adding a pyproject.toml file enables a new code path that does the build in isolation (in preparation for PEP 517; it's a good idea on it's own too) with only the build-system.requires packages available. When the build-system.requires key is missing, pip falls back to assuming it should be ["setuptools", "wheel"]. The in-development version of pip currently prints warnings when the key is not specified -- along the lines of "build-system.requires is missing" + "A future version of pip will reject pyproject.toml files that do not comply with PEP 518." and falls back to legacy behavior. Basically, pip 10 has a distinction between a missing pyproject.toml and build-system.requires = ["setuptools", "wheel"] and the PEP doesn't. However, the PEP's precise wording here would help inform the debate about how pip should behave in this edge case. I can think of at least 2 options for behavior when build-system.requires is missing: 1. Consider a missing build-system.requires equivalent to either a missing pyproject.toml or build-system.requires = ["setuptools", "wheel"]. 2. Making the build-system table mandatory in pyproject.toml. I personally think (2) would be fine -- "Explicit is better than implicit." It'll be easy to detect and error out in this case, in a way that it's possible to provide meaningful information to the user about what to do here. However, this does mean that some existing releases of projects become not-installable, which is concerning; I do think the benefits outweigh the costs though. Thoughts on this? Cheers, Pradyun
Hello, I would vote for option 2, again in the spirit of explicit better than implicit (as maintainer of tox). pip should raise an error I think and fail the build. That being said, we don't want to start breaking existing packages. So what about doing a release of pip that raises a warning first, this gives a grace period of 1 month for people to fix their packages? And then switch over to breaking. Bernat On Fri, Jun 22, 2018 at 5:34 PM Pradyun Gedam <pradyunsg@gmail.com> wrote:
Hey everyone!
In PEP 518, it is not clearly specified how a project that has a pyproject.toml file but has no build-system.requires should be treated (i.e. build-system table).
In pip 10, such a pyproject.toml file was allowed and built with setuptools and wheel, which has resulted in a lot of projects making releases that assumed that such a pyproject.toml file is valid and they use setuptools and wheel. I understand that at least pytest, towncrier and Twisted might have done so. This happened since these projects have included configuration for some tool in pyproject.toml (some of which use only pyproject.toml for configuration -- black, towncrier).
There's a little bit of subtlety here, in pip 10's implementation: adding a pyproject.toml file enables a new code path that does the build in isolation (in preparation for PEP 517; it's a good idea on it's own too) with only the build-system.requires packages available. When the build-system.requires key is missing, pip falls back to assuming it should be ["setuptools", "wheel"]. The in-development version of pip currently prints warnings when the key is not specified -- along the lines of "build-system.requires is missing" + "A future version of pip will reject pyproject.toml files that do not comply with PEP 518." and falls back to legacy behavior.
Basically, pip 10 has a distinction between a missing pyproject.toml and build-system.requires = ["setuptools", "wheel"] and the PEP doesn't. However, the PEP's precise wording here would help inform the debate about how pip should behave in this edge case.
I can think of at least 2 options for behavior when build-system.requires is missing:
1. Consider a missing build-system.requires equivalent to either a missing pyproject.toml or build-system.requires = ["setuptools", "wheel"].
2. Making the build-system table mandatory in pyproject.toml.
I personally think (2) would be fine -- "Explicit is better than implicit."
It'll be easy to detect and error out in this case, in a way that it's possible to provide meaningful information to the user about what to do here. However, this does mean that some existing releases of projects become not-installable, which is concerning; I do think the benefits outweigh the costs though.
Thoughts on this?
Cheers, Pradyun
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/7...
On Fri, 22 Jun 2018 at 09:35 Pradyun Gedam <pradyunsg@gmail.com> wrote:
Hey everyone!
In PEP 518, it is not clearly specified how a project that has a pyproject.toml file but has no build-system.requires should be treated (i.e. build-system table).
In pip 10, such a pyproject.toml file was allowed and built with setuptools and wheel, which has resulted in a lot of projects making releases that assumed that such a pyproject.toml file is valid and they use setuptools and wheel. I understand that at least pytest, towncrier and Twisted might have done so. This happened since these projects have included configuration for some tool in pyproject.toml (some of which use only pyproject.toml for configuration -- black, towncrier).
There's a little bit of subtlety here, in pip 10's implementation: adding a pyproject.toml file enables a new code path that does the build in isolation (in preparation for PEP 517; it's a good idea on it's own too) with only the build-system.requires packages available. When the build-system.requires key is missing, pip falls back to assuming it should be ["setuptools", "wheel"]. The in-development version of pip currently prints warnings when the key is not specified -- along the lines of "build-system.requires is missing" + "A future version of pip will reject pyproject.toml files that do not comply with PEP 518." and falls back to legacy behavior.
Basically, pip 10 has a distinction between a missing pyproject.toml and build-system.requires = ["setuptools", "wheel"] and the PEP doesn't. However, the PEP's precise wording here would help inform the debate about how pip should behave in this edge case.
I think the precise wording is there by not having any wording. ;) If 'requires' was meant to be optional then the PEP would have said that.
I can think of at least 2 options for behavior when build-system.requires is missing:
1. Consider a missing build-system.requires equivalent to either a missing pyproject.toml or build-system.requires = ["setuptools", "wheel"].
2. Making the build-system table mandatory in pyproject.toml.
I personally think (2) would be fine -- "Explicit is better than implicit."
And I think that's what pip's warning is saying the future will be, but they aren't quite yet ready to be that strict yet with their users (which I can understand).
It'll be easy to detect and error out in this case, in a way that it's possible to provide meaningful information to the user about what to do here. However, this does mean that some existing releases of projects become not-installable, which is concerning; I do think the benefits outweigh the costs though.
Thoughts on this?
I think what pip has planned with the warning makes sense.
On Fri, 22 Jun 2018, 22:31 Brett Cannon, <brett@python.org> wrote:
On Fri, 22 Jun 2018 at 09:35 Pradyun Gedam <pradyunsg@gmail.com> wrote:
Hey everyone!
In PEP 518, it is not clearly specified how a project that has a pyproject.toml file but has no build-system.requires should be treated (i.e. build-system table).
In pip 10, such a pyproject.toml file was allowed and built with setuptools and wheel, which has resulted in a lot of projects making releases that assumed that such a pyproject.toml file is valid and they use setuptools and wheel. I understand that at least pytest, towncrier and Twisted might have done so. This happened since these projects have included configuration for some tool in pyproject.toml (some of which use only pyproject.toml for configuration -- black, towncrier).
There's a little bit of subtlety here, in pip 10's implementation: adding a pyproject.toml file enables a new code path that does the build in isolation (in preparation for PEP 517; it's a good idea on it's own too) with only the build-system.requires packages available. When the build-system.requires key is missing, pip falls back to assuming it should be ["setuptools", "wheel"]. The in-development version of pip currently prints warnings when the key is not specified -- along the lines of "build-system.requires is missing" + "A future version of pip will reject pyproject.toml files that do not comply with PEP 518." and falls back to legacy behavior.
Basically, pip 10 has a distinction between a missing pyproject.toml and build-system.requires = ["setuptools", "wheel"] and the PEP doesn't. However, the PEP's precise wording here would help inform the debate about how pip should behave in this edge case.
I think the precise wording is there by not having any wording. ;) If 'requires' was meant to be optional then the PEP would have said that.
The PEP does say that the requires key is mandatory. Someone pointed out on the pip's tracker that it doesn't say that the build-system table is mandatory (or otherwise). And it seems that the JSON schema in the spec allows the table to be skipped entirely. That's part of the reason for this thread.
I can think of at least 2 options for behavior when build-system.requires is missing:
1. Consider a missing build-system.requires equivalent to either a missing pyproject.toml or build-system.requires = ["setuptools", "wheel"].
2. Making the build-system table mandatory in pyproject.toml.
I personally think (2) would be fine -- "Explicit is better than implicit."
And I think that's what pip's warning is saying the future will be, but they aren't quite yet ready to be that strict yet with their users (which I can understand).
It'll be easy to detect and error out in this case, in a way that it's possible to provide meaningful information to the user about what to do here. However, this does mean that some existing releases of projects become not-installable, which is concerning; I do think the benefits outweigh the costs though.
Thoughts on this?
I think what pip has planned with the warning makes sense.
On Fri, 22 Jun 2018 at 10:19 Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Fri, 22 Jun 2018, 22:31 Brett Cannon, <brett@python.org> wrote:
On Fri, 22 Jun 2018 at 09:35 Pradyun Gedam <pradyunsg@gmail.com> wrote:
Hey everyone!
In PEP 518, it is not clearly specified how a project that has a pyproject.toml file but has no build-system.requires should be treated (i.e. build-system table).
In pip 10, such a pyproject.toml file was allowed and built with setuptools and wheel, which has resulted in a lot of projects making releases that assumed that such a pyproject.toml file is valid and they use setuptools and wheel. I understand that at least pytest, towncrier and Twisted might have done so. This happened since these projects have included configuration for some tool in pyproject.toml (some of which use only pyproject.toml for configuration -- black, towncrier).
There's a little bit of subtlety here, in pip 10's implementation: adding a pyproject.toml file enables a new code path that does the build in isolation (in preparation for PEP 517; it's a good idea on it's own too) with only the build-system.requires packages available. When the build-system.requires key is missing, pip falls back to assuming it should be ["setuptools", "wheel"]. The in-development version of pip currently prints warnings when the key is not specified -- along the lines of "build-system.requires is missing" + "A future version of pip will reject pyproject.toml files that do not comply with PEP 518." and falls back to legacy behavior.
Basically, pip 10 has a distinction between a missing pyproject.toml and build-system.requires = ["setuptools", "wheel"] and the PEP doesn't. However, the PEP's precise wording here would help inform the debate about how pip should behave in this edge case.
I think the precise wording is there by not having any wording. ;) If 'requires' was meant to be optional then the PEP would have said that.
The PEP does say that the requires key is mandatory. Someone pointed out on the pip's tracker that it doesn't say that the build-system table is mandatory (or otherwise).
It says "There will be a [build-system] table in the configuration file to store build-related data". The word "will" does not suggest optional to me. ;) Once again, no one said "optional", so assume it's required.
And it seems that the JSON schema in the spec allows the table to be skipped entirely. That's part of the reason for this thread.
The JSON schema is for "illustrative purposes only", so it should not be viewed as part of the spec. IOW if you need some authoritative judgement on this you have it from one of the co-authors of the PEP. :) -Brett
I can think of at least 2 options for behavior when build-system.requires is missing:
1. Consider a missing build-system.requires equivalent to either a missing pyproject.toml or build-system.requires = ["setuptools", "wheel"].
2. Making the build-system table mandatory in pyproject.toml.
I personally think (2) would be fine -- "Explicit is better than implicit."
And I think that's what pip's warning is saying the future will be, but they aren't quite yet ready to be that strict yet with their users (which I can understand).
It'll be easy to detect and error out in this case, in a way that it's possible to provide meaningful information to the user about what to do here. However, this does mean that some existing releases of projects become not-installable, which is concerning; I do think the benefits outweigh the costs though.
Thoughts on this?
I think what pip has planned with the warning makes sense.
On Sat, 23 Jun 2018, 03:08 Brett Cannon, <brett@python.org> wrote:
On Fri, 22 Jun 2018 at 10:19 Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Fri, 22 Jun 2018, 22:31 Brett Cannon, <brett@python.org> wrote:
On Fri, 22 Jun 2018 at 09:35 Pradyun Gedam <pradyunsg@gmail.com> wrote:
Hey everyone!
In PEP 518, it is not clearly specified how a project that has a pyproject.toml file but has no build-system.requires should be treated (i.e. build-system table).
In pip 10, such a pyproject.toml file was allowed and built with setuptools and wheel, which has resulted in a lot of projects making releases that assumed that such a pyproject.toml file is valid and they use setuptools and wheel. I understand that at least pytest, towncrier and Twisted might have done so. This happened since these projects have included configuration for some tool in pyproject.toml (some of which use only pyproject.toml for configuration -- black, towncrier).
There's a little bit of subtlety here, in pip 10's implementation: adding a pyproject.toml file enables a new code path that does the build in isolation (in preparation for PEP 517; it's a good idea on it's own too) with only the build-system.requires packages available. When the build-system.requires key is missing, pip falls back to assuming it should be ["setuptools", "wheel"]. The in-development version of pip currently prints warnings when the key is not specified -- along the lines of "build-system.requires is missing" + "A future version of pip will reject pyproject.toml files that do not comply with PEP 518." and falls back to legacy behavior.
Basically, pip 10 has a distinction between a missing pyproject.toml and build-system.requires = ["setuptools", "wheel"] and the PEP doesn't. However, the PEP's precise wording here would help inform the debate about how pip should behave in this edge case.
I think the precise wording is there by not having any wording. ;) If 'requires' was meant to be optional then the PEP would have said that.
The PEP does say that the requires key is mandatory. Someone pointed out on the pip's tracker that it doesn't say that the build-system table is mandatory (or otherwise).
It says "There will be a [build-system] table in the configuration file to store build-related data". The word "will" does not suggest optional to me. ;) Once again, no one said "optional", so assume it's required.
Sounds good to me.
And it seems that the JSON schema in the spec allows the table to be skipped entirely. That's part of the reason for this thread.
The JSON schema is for "illustrative purposes only", so it should not be viewed as part of the spec.
Yeah, that's exactly what I pointed out to the user in the pip issue tracker as well.
IOW if you need some authoritative judgement on this you have it from one of the co-authors of the PEP. :)
Yeah. I think it's clear now what we have planned for pip is the right thing to do here. Thanks! Pradyun
-Brett
I can think of at least 2 options for behavior when build-system.requires is missing:
1. Consider a missing build-system.requires equivalent to either a missing pyproject.toml or build-system.requires = ["setuptools", "wheel"].
2. Making the build-system table mandatory in pyproject.toml.
I personally think (2) would be fine -- "Explicit is better than implicit."
And I think that's what pip's warning is saying the future will be, but they aren't quite yet ready to be that strict yet with their users (which I can understand).
It'll be easy to detect and error out in this case, in a way that it's possible to provide meaningful information to the user about what to do here. However, this does mean that some existing releases of projects become not-installable, which is concerning; I do think the benefits outweigh the costs though.
Thoughts on this?
I think what pip has planned with the warning makes sense.
On 23 June 2018 at 07:47, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Sat, 23 Jun 2018, 03:08 Brett Cannon, <brett@python.org> wrote:
The JSON schema is for "illustrative purposes only", so it should not be viewed as part of the spec.
Yeah, that's exactly what I pointed out to the user in the pip issue tracker as well.
Something you may want to consider is that while PEP 518 explicitly specifies that a missing pyproject.toml should be interpreted as "build-system.build-requires=['setuptools', 'wheel']", there's nothing that specifically prohibits an installer from offering a "default-for-missing-build-requires" setting. Right now, `pip` effectively defaults that hypothetical setting to "['setuptools', 'wheel']", with the proposed change being to switch it to "[]" instead (which better aligns with the intent of PEP 518). However, the new semantics mean that some sdist releases of some projects will become uninstallable. Making the installer level setting explicit rather than hypothetical would allow anyone affected to deal with that problem without either being stuck on the older version of pip, or being forced to concurrently upgrade to newer versions of the affected projects. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Jun 24, 2018 at 8:20 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sat, 23 Jun 2018, 03:08 Brett Cannon, <brett@python.org> wrote:
The JSON schema is for "illustrative purposes only", so it should not be viewed as part of the spec.
Yeah, that's exactly what I pointed out to the user in the pip issue
On 23 June 2018 at 07:47, Pradyun Gedam <pradyunsg@gmail.com> wrote: tracker
as well.
Something you may want to consider is that while PEP 518 explicitly specifies that a missing pyproject.toml should be interpreted as "build-system.build-requires=['setuptools', 'wheel']", there's nothing that specifically prohibits an installer from offering a "default-for-missing-build-requires" setting.
Right now, `pip` effectively defaults that hypothetical setting to "['setuptools', 'wheel']", with the proposed change being to switch it to "[]" instead (which better aligns with the intent of PEP 518).
However, the new semantics mean that some sdist releases of some projects will become uninstallable. Making the installer level setting explicit rather than hypothetical would allow anyone affected to deal with that problem without either being stuck on the older version of pip, or being forced to concurrently upgrade to newer versions of the affected projects.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Assuming we are going to disallow missing build-requires, I think a better way for this would be to allow a user to override build-requires on a per-package basis. It'd be a more verbose and also clearer about which packages are needing some sort of work-around to install, pushing packages to just directly specify build-requires in future releases. This is already a feature request for pip for a different use case and I think it's a reasonable request. Sure, it is extra hoops to jump through but in the long term it makes things fairly frictionless since everyone will start specifying build-system.requires -- if there's a missing build-system.requires, use the override mechanism to use setuptools and wheel (and anything else possibly) to build it. Pradyun
On 27 June 2018 at 15:59, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Sun, Jun 24, 2018 at 8:20 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
On 23 June 2018 at 07:47, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Sat, 23 Jun 2018, 03:08 Brett Cannon, <brett@python.org> wrote:
The JSON schema is for "illustrative purposes only", so it should not be viewed as part of the spec.
Yeah, that's exactly what I pointed out to the user in the pip issue tracker as well.
Something you may want to consider is that while PEP 518 explicitly specifies that a missing pyproject.toml should be interpreted as "build-system.build-requires=['setuptools', 'wheel']", there's nothing that specifically prohibits an installer from offering a "default-for-missing-build-requires" setting.
Right now, `pip` effectively defaults that hypothetical setting to "['setuptools', 'wheel']", with the proposed change being to switch it to "[]" instead (which better aligns with the intent of PEP 518).
However, the new semantics mean that some sdist releases of some projects will become uninstallable. Making the installer level setting explicit rather than hypothetical would allow anyone affected to deal with that problem without either being stuck on the older version of pip, or being forced to concurrently upgrade to newer versions of the affected projects.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Assuming we are going to disallow missing build-requires, I think a better way for this would be to allow a user to override build-requires on a per-package basis. It'd be a more verbose and also clearer about which packages are needing some sort of work-around to install, pushing packages to just directly specify build-requires in future releases.
I'm not sure what you're suggesting here (but I suspect it's a pip-specific implementation detail rather than a standards issue). But there are three separate cases to consider: 1. Project has pyproject.toml with build-system.requires specified. No problem, full PEP 518 behaviour (and pip uses build isolation). 2. Project has pyproject.toml but no build-system.requires, Illegal, confirmed by this discussion, so terminate with an error (pip currenly warns, but will move to an error after a deprecation period). 3. Project has no pyproject.toml. Old-style project, PEP 518 says that the default is [setuptools, wheel]. Pip will actually use the legacy (no build isolation) code path, which is a backward compatibility choice. I'm not actually sure PEP 518 needs to even comment on this case, as such projects clearly don't conform to that PEP and tools like pip will of necessity have to handle them as "legacy". The only case where I can see your "per-package overrides" fitting in would be (3), which is outside the scope of PEP 518 and so really a pip-only issue. But if you're saying that the end user, when installing (from source) a package that has no pyproject.toml, can say something like "instead of just installing setuptools and wheel, install A, B, anc C", then a) A, B and C had better include setuptools and wheel, or we're hosed, b) We're on the legacy path, so we're not actually using build isolation (and so we're not able to install anything), c) I'm not sure I see why the end user would want to override what the project expected anyway. I'm pretty sure I've missed something here. I suggest we move this discussion to a feature request on the pip tracker, where you can explain your proposal in a bit more detail.
This is already a feature request for pip for a different use case and I think it's a reasonable request.
What's the issue number? I don't recall seeing anything like this.
Sure, it is extra hoops to jump through but in the long term it makes things fairly frictionless since everyone will start specifying build-system.requires -- if there's a missing build-system.requires, use the override mechanism to use setuptools and wheel (and anything else possibly) to build it.
I don't see how this (an end user feature) will encourage projects to move to using pyproject.toml - it seems like the opposite is more likely. Paul
On Wed, Jun 27, 2018 at 9:15 PM Paul Moore <p.f.moore@gmail.com> wrote:
On 27 June 2018 at 15:59, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Sun, Jun 24, 2018 at 8:20 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
On 23 June 2018 at 07:47, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Sat, 23 Jun 2018, 03:08 Brett Cannon, <brett@python.org> wrote:
The JSON schema is for "illustrative purposes only", so it should not be viewed as part of the spec.
Yeah, that's exactly what I pointed out to the user in the pip issue tracker as well.
Something you may want to consider is that while PEP 518 explicitly specifies that a missing pyproject.toml should be interpreted as "build-system.build-requires=['setuptools', 'wheel']", there's nothing that specifically prohibits an installer from offering a "default-for-missing-build-requires" setting.
Right now, `pip` effectively defaults that hypothetical setting to "['setuptools', 'wheel']", with the proposed change being to switch it to "[]" instead (which better aligns with the intent of PEP 518).
However, the new semantics mean that some sdist releases of some projects will become uninstallable. Making the installer level setting explicit rather than hypothetical would allow anyone affected to deal with that problem without either being stuck on the older version of pip, or being forced to concurrently upgrade to newer versions of the affected projects.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Assuming we are going to disallow missing build-requires, I think a better way for this would be to allow a user to override build-requires on a per-package basis. It'd be a more verbose and also clearer about which packages are needing some sort of work-around to install, pushing packages to just directly specify build-requires in future releases.
I'm not sure what you're suggesting here (but I suspect it's a pip-specific implementation detail rather than a standards issue).
Now that I think about it, yeah. Sorry for the noise!
But there are three separate cases to consider:
1. Project has pyproject.toml with build-system.requires specified. No problem, full PEP 518 behaviour (and pip uses build isolation). 2. Project has pyproject.toml but no build-system.requires, Illegal, confirmed by this discussion, so terminate with an error (pip currenly warns, but will move to an error after a deprecation period). 3. Project has no pyproject.toml. Old-style project, PEP 518 says that the default is [setuptools, wheel]. Pip will actually use the legacy (no build isolation) code path, which is a backward compatibility choice. I'm not actually sure PEP 518 needs to even comment on this case, as such projects clearly don't conform to that PEP and tools like pip will of necessity have to handle them as "legacy".
The only case where I can see your "per-package overrides" fitting in would be (3), which is outside the scope of PEP 518 and so really a pip-only issue.
I am suggesting it for (2). I'll elaborate on the tracker. But if you're saying that the end user, when
installing (from source) a package that has no pyproject.toml, can say something like "instead of just installing setuptools and wheel, install A, B, anc C", then
a) A, B and C had better include setuptools and wheel, or we're hosed, b) We're on the legacy path, so we're not actually using build isolation (and so we're not able to install anything), c) I'm not sure I see why the end user would want to override what the project expected anyway.
I'm pretty sure I've missed something here. I suggest we move this discussion to a feature request on the pip tracker, where you can explain your proposal in a bit more detail.
Indeed.
This is already a feature request for pip for a different use case and I think it's a reasonable request.
What's the issue number? I don't recall seeing anything like this.
https://github.com/pypa/pip/issues/4582
Sure, it is extra hoops to jump through but in the long term it
makes things fairly frictionless since everyone will start specifying build-system.requires -- if there's a missing build-system.requires, use the override mechanism to use setuptools and wheel (and anything else possibly) to build it.
I don't see how this (an end user feature) will encourage projects to move to using pyproject.toml - it seems like the opposite is more likely.
Paul
On 27 June 2018 at 22:09, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Wed, Jun 27, 2018 at 9:15 PM Paul Moore <p.f.moore@gmail.com> wrote:
On 27 June 2018 at 15:59, Pradyun Gedam <pradyunsg@gmail.com> wrote:
Assuming we are going to disallow missing build-requires, I think a better way for this would be to allow a user to override build-requires on a per-package basis. It'd be a more verbose and also clearer about which packages are needing some sort of work-around to install, pushing packages to just directly specify build-requires in future releases.
[...] 1. Project has pyproject.toml with build-system.requires specified. No problem, full PEP 518 behaviour (and pip uses build isolation). 2. Project has pyproject.toml but no build-system.requires, Illegal, confirmed by this discussion, so terminate with an error (pip currenly warns, but will move to an error after a deprecation period). 3. Project has no pyproject.toml. Old-style project, PEP 518 says that the default is [setuptools, wheel]. Pip will actually use the legacy (no build isolation) code path, which is a backward compatibility choice. I'm not actually sure PEP 518 needs to even comment on this case, as such projects clearly don't conform to that PEP and tools like pip will of necessity have to handle them as "legacy".
The only case where I can see your "per-package overrides" fitting in would be (3), which is outside the scope of PEP 518 and so really a pip-only issue.
I am suggesting it for (2).
But Brett clearly stated that he views PEP 518 as stating that the build-system.requires key in pyproject.toml is *not* optional. And I think that's the correct reading of the PEP. Projects that want to use pyproject.toml for tool settings can, and should, just set build-system.requires (presumably to ['setuptools', 'wheel'] as they obviously don't need a non-default value).
I'll elaborate on the tracker.
OK. But I'm going to take the view that having explicitly requested clarification on distutils-sig, if we want to do anything other than reject a pyproject.toml with a missing build-system.requires as invalid, we need to first of all get the PEP changed. And for that we'll have to have a good use case to justify the change. Personally, I don't see the value.
This is already a feature request for pip for a different use case and I think it's a reasonable request.
What's the issue number? I don't recall seeing anything like this.
I found that discussion confusing at the time, and on rereading I don't think my view has changed :-( I'm going to be offline for a week or two, but if you do raise a tracker item, I'll pick it up once I return. Paul
On Wed, Jun 27, 2018 at 2:25 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 27 June 2018 at 22:09, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Wed, Jun 27, 2018 at 9:15 PM Paul Moore <p.f.moore@gmail.com> wrote:
On 27 June 2018 at 15:59, Pradyun Gedam <pradyunsg@gmail.com> wrote:
Assuming we are going to disallow missing build-requires, I think a better way for this would be to allow a user to override build-requires on a per-package basis. It'd be a more verbose and also clearer about which packages are needing some sort of work-around to install, pushing packages to just directly specify build-requires in future releases.
[...] 1. Project has pyproject.toml with build-system.requires specified. No problem, full PEP 518 behaviour (and pip uses build isolation). 2. Project has pyproject.toml but no build-system.requires, Illegal, confirmed by this discussion, so terminate with an error (pip currenly warns, but will move to an error after a deprecation period). 3. Project has no pyproject.toml. Old-style project, PEP 518 says that the default is [setuptools, wheel]. Pip will actually use the legacy (no build isolation) code path, which is a backward compatibility choice. I'm not actually sure PEP 518 needs to even comment on this case, as such projects clearly don't conform to that PEP and tools like pip will of necessity have to handle them as "legacy".
The only case where I can see your "per-package overrides" fitting in would be (3), which is outside the scope of PEP 518 and so really a pip-only issue.
I am suggesting it for (2).
But Brett clearly stated that he views PEP 518 as stating that the build-system.requires key in pyproject.toml is *not* optional. And I think that's the correct reading of the PEP.
FWIW, I'm a co-author on the PEP, and I've apparently been shipping software without this key for most of a year: https://github.com/python-trio/trio/blob/master/pyproject.toml Clearly I should have read my PEP more closely...
OK. But I'm going to take the view that having explicitly requested clarification on distutils-sig, if we want to do anything other than reject a pyproject.toml with a missing build-system.requires as invalid, we need to first of all get the PEP changed. And for that we'll have to have a good use case to justify the change. Personally, I don't see the value.
If pip starts erroring out when pyproject.toml is missing build-system.requires, then 'pip install twisted' will stop working and everyone will start screaming on twitter: https://github.com/twisted/twisted/blob/trunk/pyproject.toml (Trio won't actually be affected b/c it has a universal wheel, so pip always uses that instead of the sdist. But twisted only has wheels for python 2 on Windows; all other configurations build from source, and thus will break.) We probably need to update the PEPs some here anyway, if only for clarity. Also, right now PEP 517 says "If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should fall back to running setup.py." But thinking about it again, we probably don't want to do this, because it adds Yet Another build configuration that pip has to handle: 1. no pyproject.toml -> legacy non-isolated build via legacy setup.py support 2. pyproject.toml without build-backend specified -> new isolated build via legacy setup.py support 3. pyproject.toml with build-backend specified -> new isolated build via new build-backend support Once pip and setuptools both have PEP 517 build backend support (which should hopefully happen soon?), then option (2) will become weird and unnecessary. It would be nice to get rid of it. So I think we'll be having a similar discussion in a few months about how to handle pyproject.toml without build-backend keys. At that point we'll have a few options: - We could say that build-system.build-backend is mandatory, and error out if it's missing. But that will break the world (probably even more so than making build-system.requires mandatory now will break the world, since by the time this happens there will be more pyproject.toml files in the wild). - We could shrug and say that updating PEPs is a lot of work so let's just go with what we wrote way back when. But that forces pip to forever carry around code to implement both option (2) and (3), which do essentially the same thing except with extra code and probably there will be weird bugs that show up in one configuration or the other, etc. - We could say that when build-system.build-backend is missing, it defaults to "setuptools". But it's *really weird* to have a default value for build-system.build-backend while not having one for build-system.requires, because, effectively, the interpretation of build-system.build-backend depends on the value of build-system.requires. The build backend is an object that gets looked up inside those required packages. If we're going to be like "we have no idea what setuptools is, you have to tell us to install it every time", then it's very weird to turn around in the next line like "oh yeah we *use* setuptools by default, we just don't *install* it by default". So my inclination is to plan on ending up with build-system.requires defaulting to ["setuptools", "wheel"], and build-system.backend defaulting to "setuptools". Hopefully we'll eventually get to a place where ~no-one uses these defaults, but carrying around the code to handle the defaults isn't really a burden.
This is already a feature request for pip for a different use case and I think it's a reasonable request.
What's the issue number? I don't recall seeing anything like this.
I found that discussion confusing at the time, and on rereading I don't think my view has changed :-(
That discussion is indeed pretty confusing, but the basic idea of letting the user override build requires is easy to motivate. Say you're trying to install 'foomatic v1.2', which has a build requirement of "setuptools". But oops, setuptools v45.0 was just released, and it turns out that foomatic v1.2 depends on some internal implementation details of setuptools that changed in this release. That's annoying. Probably you should file a bug, and the foomatic authors will fix it and eventually make a new release. But that doesn't help you *right now*, when you're trying to get out an emergency deploy and your boss is breathing down your neck and you just need the stupid foomatic v1.2 to install so you can get on with things. Wouldn't it be nice (tm), if you could do: pip install foomatic==1.2 --force-build-requirement='setuptools<45' -n -- Nathaniel J. Smith -- https://vorpus.org
On Thu, 28 Jun 2018, 04:38 Nathaniel Smith, <njs@pobox.com> wrote:
- We could say that build-system.build-backend is mandatory, and error out if it's missing. But that will break the world (probably even more so than making build-system.requires mandatory now will break the world, since by the time this happens there will be more pyproject.toml files in the wild). [...]
So my inclination is to plan on ending up with build-system.requires
defaulting to ["setuptools", "wheel"], and build-system.backend defaulting to "setuptools". Hopefully we'll eventually get to a place where ~no-one uses these defaults, but carrying around the code to handle the defaults isn't really a burden.
In the second option - we might never get to the place where no-one uses it because, why would anyone add the boilerplate if not writing it achieves the desired result? Another option exists and I think that's what pradysung was suggesting - error out (after the grace period), and when something breaks direct people to use the --force-build-requires=setuptools flag to fix it "Right Now" and file a ticket with the package to get the "Real Fix". And make the flag work on a per-package basis (like the only-binary flag) The same can be done with build-backend (PEP will need to change to disallow it missing)
On 28 June 2018 at 02:37, Nathaniel Smith <njs@pobox.com> wrote:
On Wed, Jun 27, 2018 at 2:25 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 27 June 2018 at 22:09, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Wed, Jun 27, 2018 at 9:15 PM Paul Moore <p.f.moore@gmail.com> wrote:
But Brett clearly stated that he views PEP 518 as stating that the build-system.requires key in pyproject.toml is *not* optional. And I think that's the correct reading of the PEP.
FWIW, I'm a co-author on the PEP, and I've apparently been shipping software without this key for most of a year: https://github.com/python-trio/trio/blob/master/pyproject.toml Clearly I should have read my PEP more closely...
Oh! I'm really sorry - I thought you'd seen the discussion and the fact that you hadn't responded to Brett's assertion directly meant you were OK with it. I think I must have been skimming at the time and not fully understood the point you were making in your post. My error, and thanks for making me aware of it.
OK. But I'm going to take the view that having explicitly requested clarification on distutils-sig, if we want to do anything other than reject a pyproject.toml with a missing build-system.requires as invalid, we need to first of all get the PEP changed. And for that we'll have to have a good use case to justify the change. Personally, I don't see the value.
If pip starts erroring out when pyproject.toml is missing build-system.requires, then 'pip install twisted' will stop working and everyone will start screaming on twitter: https://github.com/twisted/twisted/blob/trunk/pyproject.toml
(Trio won't actually be affected b/c it has a universal wheel, so pip always uses that instead of the sdist. But twisted only has wheels for python 2 on Windows; all other configurations build from source, and thus will break.)
I can see the practicality argument here. I don't think (correct me if I'm wrong!) that anyone particularly anticipated when we were drafting PEP 518 that the tools section would be used by anything other than build tools - I know I hadn't anticipated that usage. In hindsight it's an obvious thing to do, but it does mean that pyproject.toml is more of a "general project configuration" file than purely a build configuration one.
We probably need to update the PEPs some here anyway, if only for clarity.
Agreed
Also, right now PEP 517 says "If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should fall back to running setup.py." But thinking about it again, we probably don't want to do this, because it adds Yet Another build configuration that pip has to handle:
1. no pyproject.toml -> legacy non-isolated build via legacy setup.py support 2. pyproject.toml without build-backend specified -> new isolated build via legacy setup.py support 3. pyproject.toml with build-backend specified -> new isolated build via new build-backend support
Once pip and setuptools both have PEP 517 build backend support (which should hopefully happen soon?), then option (2) will become weird and unnecessary. It would be nice to get rid of it. So I think we'll be having a similar discussion in a few months about how to handle pyproject.toml without build-backend keys. At that point we'll have a few options:
- We could say that build-system.build-backend is mandatory, and error out if it's missing. But that will break the world (probably even more so than making build-system.requires mandatory now will break the world, since by the time this happens there will be more pyproject.toml files in the wild).
- We could shrug and say that updating PEPs is a lot of work so let's just go with what we wrote way back when. But that forces pip to forever carry around code to implement both option (2) and (3), which do essentially the same thing except with extra code and probably there will be weird bugs that show up in one configuration or the other, etc.
- We could say that when build-system.build-backend is missing, it defaults to "setuptools". But it's *really weird* to have a default value for build-system.build-backend while not having one for build-system.requires, because, effectively, the interpretation of build-system.build-backend depends on the value of build-system.requires. The build backend is an object that gets looked up inside those required packages. If we're going to be like "we have no idea what setuptools is, you have to tell us to install it every time", then it's very weird to turn around in the next line like "oh yeah we *use* setuptools by default, we just don't *install* it by default".
So my inclination is to plan on ending up with build-system.requires defaulting to ["setuptools", "wheel"], and build-system.backend defaulting to "setuptools". Hopefully we'll eventually get to a place where ~no-one uses these defaults, but carrying around the code to handle the defaults isn't really a burden.
That's certainly the other option that we'd want to follow if the keys aren't mandatory. Focusing back on the PEP 518 side of things, the biggest problem is that the behaviour under build-system.requires = ["setuptools", "wheel"] is somewhat different from the legacy code path (although I'd have to do some research to say exactly how - at the moment things we proposed to do, things we plan on doing, and things we actually do right now, are hopelessly muddled in my mind) and I think that switching mode triggered by the existence of pyproject.toml has caused problems for some people (although that may only be because of build isolation, and the --no-build-isolation flag covers that case, albeit by making it the end user's problem rather than the project's). But I don't have a strong opinion either way here - that's why I originally suggested bringing the discussion to distutils-sig. I'm happy for pip to go with whatever the consensus ends up being. And whatever way it goes, a PEP update to make the intent explicit would probably be worthwhile.
This is already a feature request for pip for a different use case and I think it's a reasonable request.
What's the issue number? I don't recall seeing anything like this.
I found that discussion confusing at the time, and on rereading I don't think my view has changed :-(
That discussion is indeed pretty confusing, but the basic idea of letting the user override build requires is easy to motivate.
Say you're trying to install 'foomatic v1.2', which has a build requirement of "setuptools". But oops, setuptools v45.0 was just released, and it turns out that foomatic v1.2 depends on some internal implementation details of setuptools that changed in this release. That's annoying. Probably you should file a bug, and the foomatic authors will fix it and eventually make a new release. But that doesn't help you *right now*, when you're trying to get out an emergency deploy and your boss is breathing down your neck and you just need the stupid foomatic v1.2 to install so you can get on with things. Wouldn't it be nice (tm), if you could do:
pip install foomatic==1.2 --force-build-requirement='setuptools<45'
Hmm, OK. For that scenario, I'd argue the user can just say --no-build-isolation and set up the build environment manually. There's a risk here that we're creating ever more complicated mechanisms for relatively niche corner cases[1], and I think we need to be very careful to examine the trade-offs between code (and design) complexity and overall utility. The use of pip in development workflows is already a huge source of complexity (people have incredibly diverse workflows, which they typically cannot describe very well, because they often have no idea how unusual what they are doing can be). We have to cater for at least some of that, but (with my pip maintainer hat on) I think we need to draw the line somewhere, if only because of lack of manpower. Paul [1] From my POV, pip building from source is already a relatively niche case - and even if that's not the reality today for some users, I remain pretty firmly convinced that it's what we should be aiming for.
I don't consider building from source a niche use case per se, given that pip is most widely adopted PEP-517/518 build frontend. What other build frontends there are or are to be recommended for usage (with setuptools/flint/poetry being mostly backends)? On Thu, Jun 28, 2018 at 8:35 AM Paul Moore <p.f.moore@gmail.com> wrote:
On 28 June 2018 at 02:37, Nathaniel Smith <njs@pobox.com> wrote:
On Wed, Jun 27, 2018 at 2:25 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 27 June 2018 at 22:09, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Wed, Jun 27, 2018 at 9:15 PM Paul Moore <p.f.moore@gmail.com>
wrote:
But Brett clearly stated that he views PEP 518 as stating that the build-system.requires key in pyproject.toml is *not* optional. And I think that's the correct reading of the PEP.
FWIW, I'm a co-author on the PEP, and I've apparently been shipping software without this key for most of a year: https://github.com/python-trio/trio/blob/master/pyproject.toml Clearly I should have read my PEP more closely...
Oh! I'm really sorry - I thought you'd seen the discussion and the fact that you hadn't responded to Brett's assertion directly meant you were OK with it. I think I must have been skimming at the time and not fully understood the point you were making in your post. My error, and thanks for making me aware of it.
OK. But I'm going to take the view that having explicitly requested clarification on distutils-sig, if we want to do anything other than reject a pyproject.toml with a missing build-system.requires as invalid, we need to first of all get the PEP changed. And for that we'll have to have a good use case to justify the change. Personally, I don't see the value.
If pip starts erroring out when pyproject.toml is missing build-system.requires, then 'pip install twisted' will stop working and everyone will start screaming on twitter: https://github.com/twisted/twisted/blob/trunk/pyproject.toml
(Trio won't actually be affected b/c it has a universal wheel, so pip always uses that instead of the sdist. But twisted only has wheels for python 2 on Windows; all other configurations build from source, and thus will break.)
I can see the practicality argument here. I don't think (correct me if I'm wrong!) that anyone particularly anticipated when we were drafting PEP 518 that the tools section would be used by anything other than build tools - I know I hadn't anticipated that usage. In hindsight it's an obvious thing to do, but it does mean that pyproject.toml is more of a "general project configuration" file than purely a build configuration one.
We probably need to update the PEPs some here anyway, if only for clarity.
Agreed
Also, right now PEP 517 says "If the pyproject.toml file is absent, or the build-backend key is missing, the source tree is not using this specification, and tools should fall back to running setup.py." But thinking about it again, we probably don't want to do this, because it adds Yet Another build configuration that pip has to handle:
1. no pyproject.toml -> legacy non-isolated build via legacy setup.py support 2. pyproject.toml without build-backend specified -> new isolated build via legacy setup.py support 3. pyproject.toml with build-backend specified -> new isolated build via new build-backend support
Once pip and setuptools both have PEP 517 build backend support (which should hopefully happen soon?), then option (2) will become weird and unnecessary. It would be nice to get rid of it. So I think we'll be having a similar discussion in a few months about how to handle pyproject.toml without build-backend keys. At that point we'll have a few options:
- We could say that build-system.build-backend is mandatory, and error out if it's missing. But that will break the world (probably even more so than making build-system.requires mandatory now will break the world, since by the time this happens there will be more pyproject.toml files in the wild).
- We could shrug and say that updating PEPs is a lot of work so let's just go with what we wrote way back when. But that forces pip to forever carry around code to implement both option (2) and (3), which do essentially the same thing except with extra code and probably there will be weird bugs that show up in one configuration or the other, etc.
- We could say that when build-system.build-backend is missing, it defaults to "setuptools". But it's *really weird* to have a default value for build-system.build-backend while not having one for build-system.requires, because, effectively, the interpretation of build-system.build-backend depends on the value of build-system.requires. The build backend is an object that gets looked up inside those required packages. If we're going to be like "we have no idea what setuptools is, you have to tell us to install it every time", then it's very weird to turn around in the next line like "oh yeah we *use* setuptools by default, we just don't *install* it by default".
So my inclination is to plan on ending up with build-system.requires defaulting to ["setuptools", "wheel"], and build-system.backend defaulting to "setuptools". Hopefully we'll eventually get to a place where ~no-one uses these defaults, but carrying around the code to handle the defaults isn't really a burden.
That's certainly the other option that we'd want to follow if the keys aren't mandatory. Focusing back on the PEP 518 side of things, the biggest problem is that the behaviour under build-system.requires = ["setuptools", "wheel"] is somewhat different from the legacy code path (although I'd have to do some research to say exactly how - at the moment things we proposed to do, things we plan on doing, and things we actually do right now, are hopelessly muddled in my mind) and I think that switching mode triggered by the existence of pyproject.toml has caused problems for some people (although that may only be because of build isolation, and the --no-build-isolation flag covers that case, albeit by making it the end user's problem rather than the project's).
But I don't have a strong opinion either way here - that's why I originally suggested bringing the discussion to distutils-sig. I'm happy for pip to go with whatever the consensus ends up being. And whatever way it goes, a PEP update to make the intent explicit would probably be worthwhile.
This is already a feature request for pip for a different use case and I think it's a reasonable request.
What's the issue number? I don't recall seeing anything like this.
I found that discussion confusing at the time, and on rereading I don't think my view has changed :-(
That discussion is indeed pretty confusing, but the basic idea of letting the user override build requires is easy to motivate.
Say you're trying to install 'foomatic v1.2', which has a build requirement of "setuptools". But oops, setuptools v45.0 was just released, and it turns out that foomatic v1.2 depends on some internal implementation details of setuptools that changed in this release. That's annoying. Probably you should file a bug, and the foomatic authors will fix it and eventually make a new release. But that doesn't help you *right now*, when you're trying to get out an emergency deploy and your boss is breathing down your neck and you just need the stupid foomatic v1.2 to install so you can get on with things. Wouldn't it be nice (tm), if you could do:
pip install foomatic==1.2 --force-build-requirement='setuptools<45'
Hmm, OK. For that scenario, I'd argue the user can just say --no-build-isolation and set up the build environment manually. There's a risk here that we're creating ever more complicated mechanisms for relatively niche corner cases[1], and I think we need to be very careful to examine the trade-offs between code (and design) complexity and overall utility.
The use of pip in development workflows is already a huge source of complexity (people have incredibly diverse workflows, which they typically cannot describe very well, because they often have no idea how unusual what they are doing can be). We have to cater for at least some of that, but (with my pip maintainer hat on) I think we need to draw the line somewhere, if only because of lack of manpower.
Paul
[1] From my POV, pip building from source is already a relatively niche case - and even if that's not the reality today for some users, I remain pretty firmly convinced that it's what we should be aiming for. -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/Q...
On 28 June 2018 at 09:55, Bernat Gabor <gaborjbernat@gmail.com> wrote:
I don't consider building from source a niche use case per se, given that pip is most widely adopted PEP-517/518 build frontend. What other build frontends there are or are to be recommended for usage (with setuptools/flint/poetry being mostly backends)?
The vast majority of uses of pip are `pip install xxx` which will typically download and install a wheel from PyPI. Those just work. Consumers of packages vastly outnumber producers. Use of pip to build packages for publication is an important use case, but (a) it's definitely not the most significant case and (b) it works pretty well. The biggest issues seem to be with use of pip in local development of applications and packages that will never be published on PyPI ("we use pip to install packages from our git repository, and our various tools have complex dependencies - how do we make a dependency on build-123 of package foo from git?") and yet those are far in the minority of actual use. Basically it's the usual 80/20 rule - 80% of the problems come from 20% of the use cases - but exaggerated drastically (I suspect it's 99/1 or worse, in reality). Paul PS The biggest outlier here is the scientific stack, which has significant build challenges, and yet enabling them to robustly publish wheels is a critical use case.
On Thu, Jun 28, 2018, at 8:25 AM, Paul Moore wrote:
I can see the practicality argument here. I don't think (correct me if I'm wrong!) that anyone particularly anticipated when we were drafting PEP 518 that the tools section would be used by anything other than build tools - I know I hadn't anticipated that usage. In hindsight it's an obvious thing to do, but it does mean that pyproject.toml is more of a "general project configuration" file than purely a build configuration one.
I agree - I didn't really think about this, but it is obvious in hindsight: we provide one obvious place to put configuration, and people put it there. :-) Maybe pip should be switching behaviour based on the presence of the [build-system] table in the toml file, rather than just the presence of pyproject.toml? (I'm only half following this discussion, so apologies if someone already suggested this). Thomas
On Thu, Jun 28, 2018, 14:00 Thomas Kluyver, <thomas@kluyver.me.uk> wrote:
On Thu, Jun 28, 2018, at 8:25 AM, Paul Moore wrote:
I can see the practicality argument here. I don't think (correct me if I'm wrong!) that anyone particularly anticipated when we were drafting PEP 518 that the tools section would be used by anything other than build tools - I know I hadn't anticipated that usage. In hindsight it's an obvious thing to do, but it does mean that pyproject.toml is more of a "general project configuration" file than purely a build configuration one.
I agree - I didn't really think about this, but it is obvious in hindsight: we provide one obvious place to put configuration, and people put it there. :-)
The file was originally meant to target building wheels for libraries. It just happens that folks have pushed that out to include apps as well. So if the purpose of the file expands to include apps as well then that changes what the PEP should require to be in the file.
Maybe pip should be switching behaviour based on the presence of the [build-system] table in the toml file, rather than just the presence of pyproject.toml?
If the decision is to make having the value optional then I agree there should be 2 code paths: isolated based on pyproject.toml and build requirements specified, or the old way. I would also consider continuing to issue a warning for the lack of details in pyproject.toml (and maybe lacking pyproject.toml some day?).
(I'm only half following this discussion, so apologies if someone already suggested this).
Thomas -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/H...
The file was originally meant to target building wheels for libraries. It just happens that folks have pushed that out to include apps as well. So if the purpose of the file expands to include apps as well then that changes what the PEP should require to be in the file. It even expanded to include tools not directly related to packaging at all - towncrier was given as an example, and that's a developer tool for
On Thu, Jun 28, 2018, at 6:27 PM, Brett Cannon wrote: producing release notes. I guess we implicitly invited this generalisation by calling it 'pyproject.toml' rather than anything explicitly tied to building or packaging. In any case, I think it's too late to say such tools shouldn't use it, and as Nathaniel pointed out, it's pretty surprising that using a tool like towncrier would suddenly change how pip deals with your package. So I think the best option is to say that the presence or absence of pyproject.toml is not significant for build frontends - rather it's the presence of build-system information in that file that matters. Thomas
In the pep it's stated tools can use the tool section https://www.python.org/dev/peps/pep-0518/#id28 and at no point says build tools only. So I don't think at all strange that towncrier uses it. It follows the words of the pep quite rigourously. On Thu, Jun 28, 2018, 18:40 Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Thu, Jun 28, 2018, at 6:27 PM, Brett Cannon wrote:
The file was originally meant to target building wheels for libraries. It just happens that folks have pushed that out to include apps as well. So if the purpose of the file expands to include apps as well then that changes what the PEP should require to be in the file.
It even expanded to include tools not directly related to packaging at all - towncrier was given as an example, and that's a developer tool for producing release notes. I guess we implicitly invited this generalisation by calling it 'pyproject.toml' rather than anything explicitly tied to building or packaging.
In any case, I think it's too late to say such tools shouldn't use it, and as Nathaniel pointed out, it's pretty surprising that using a tool like towncrier would suddenly change how pip deals with your package. So I think the best option is to say that the presence or absence of pyproject.toml is not significant for build frontends - rather it's the presence of build-system information in that file that matters.
Thomas -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/C...
On 28 June 2018 at 18:45, Bernat Gabor <gaborjbernat@gmail.com> wrote:
In the pep it's stated tools can use the tool section https://www.python.org/dev/peps/pep-0518/#id28 and at no point says build tools only. So I don't think at all strange that towncrier uses it. It follows the words of the pep quite rigourously.
The whole PEP is about *build* tools, and in that context, that's what was meant. Maybe the PEP should have been more explicit. Maybe we should have thought about this at the time. Maybe lots of things, but the reality is that we *didn't* intend it to be used for non-build tools, and nevertheless it is being used in that way. We should deal with things as they are now, and not spend ages debating whether things "should" be the way they are. That's not to say that build-system.requires *has* to be optional - we could decide that we want to be strict and towncrier et al have to deal with it. But it does mean that doing that will cause breakage, which I suspect means that we shouldn't. Paul
On Thu, Jun 28, 2018 at 11:19 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 28 June 2018 at 18:45, Bernat Gabor <gaborjbernat@gmail.com> wrote:
In the pep it's stated tools can use the tool section https://www.python.org/dev/peps/pep-0518/#id28 and at no point says build tools only. So I don't think at all strange that towncrier uses it. It follows the words of the pep quite rigourously.
The whole PEP is about *build* tools, and in that context, that's what was meant. Maybe the PEP should have been more explicit. Maybe we should have thought about this at the time. Maybe lots of things, but the reality is that we *didn't* intend it to be used for non-build tools, and nevertheless it is being used in that way. We should deal with things as they are now, and not spend ages debating whether things "should" be the way they are.
Oh, I totally imagined that it would be used for non-build tools. I was thinking of stuff like pytest when I wrote that 'tool namespace' section, and of the way all kinds of non-build tools had already colonized setup.cfg (cf. Donald's "paving the cow paths"). It's the One Obvious place for project-specific configuration. That's awesome! Python packaging needs more One Obvious things. -n -- Nathaniel J. Smith -- https://vorpus.org
On Thu, Jun 28, 2018, 14:38 Thomas Kluyver, <thomas@kluyver.me.uk> wrote:
On Thu, Jun 28, 2018, at 6:27 PM, Brett Cannon wrote:
The file was originally meant to target building wheels for libraries. It just happens that folks have pushed that out to include apps as well. So if the purpose of the file expands to include apps as well then that changes what the PEP should require to be in the file.
It even expanded to include tools not directly related to packaging at all - towncrier was given as an example, and that's a developer tool for producing release notes. I guess we implicitly invited this generalisation by calling it 'pyproject.toml' rather than anything explicitly tied to building or packaging.
I brought this up earlier but towncrier and Black can both be viewed as build tools as they help prepare your library for the final wheel. IOW since they help prep your code release they have a place in the file.
In any case, I think it's too late to say such tools shouldn't use it, and as Nathaniel pointed out, it's pretty surprising that using a tool like towncrier would suddenly change how pip deals with your package. So I think the best option is to say that the presence or absence of pyproject.toml is not significant for build frontends - rather it's the presence of build-system information in that file that matters.
I didn't mean to suggest that tools shouldn't use the file (we added tool namespaces for a reason which you already use 😁). All I'm saying is that originally the goal was for building libraries, which means not specifying build tools didn't really make sense. As I said, if the decision is the community wants to take the file in another direction and we want to open it up for apps as well then taking your suggestion to make the entire build section optional but its existence requires providing all metadata then that seems reasonable. +Brett
Thomas
On 28 June 2018 at 11:37, Nathaniel Smith <njs@pobox.com> wrote:
So my inclination is to plan on ending up with build-system.requires defaulting to ["setuptools", "wheel"], and build-system.backend defaulting to "setuptools". Hopefully we'll eventually get to a place where ~no-one uses these defaults, but carrying around the code to handle the defaults isn't really a burden.
While I was going to post to say I liked this approach, after a bit of reflection, I realised I prefer Thomas Kluyver's suggestion: instead of taking "pyproject.toml" as indicating a build-isolation compatible sdist file, instead make "pyproject.toml with a build-system table" the marker for that case. If you don't have a build-system table at all, then you'll continue to get the legacy sdist handling, allowing the addition of other tool config without impacting the way your sdist gets built. If you do add a build-system table, then you have to populates the "requires" field properly, even if you're using setuptools as your build backend. That way, the "build-system.backend defaults to setuptools" behaviour is only there to support pyproject.toml files that have already opted in to build isolation by writing: [build-system] requires = ["setuptools", "wheel"] Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 30 June 2018 at 06:33, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 June 2018 at 11:37, Nathaniel Smith <njs@pobox.com> wrote:
So my inclination is to plan on ending up with build-system.requires defaulting to ["setuptools", "wheel"], and build-system.backend defaulting to "setuptools". Hopefully we'll eventually get to a place where ~no-one uses these defaults, but carrying around the code to handle the defaults isn't really a burden.
While I was going to post to say I liked this approach, after a bit of reflection, I realised I prefer Thomas Kluyver's suggestion: instead of taking "pyproject.toml" as indicating a build-isolation compatible sdist file, instead make "pyproject.toml with a build-system table" the marker for that case.
As far as I can see, the only difference this makes is that it means pip retains the legacy (non-isolated) behaviour in a few more places (specifically places where it's quite likley the project hasn't thought about build isolation). So it's basically a slightly more forgiving version of Nathaniel's proposal. The part of Nathaniel's approach that I think would be most confusing is a project that currently uses setup_requires which adds a pyproject.toml for (say) towncrier. The build would become isolated, but setup_requires (which is implemented by setuptools, not pip) would ignore the isolated environment and install in the wrong place (maybe? I honestly don't know). I'm quite happy to call this deprecated behaviour and point out that the project should switch to explicitly using PEP 518, but given that this whole discussion is because people haven't done that, I suspect Nathaniel's proposal doesn't actually solve the root issue here...
If you don't have a build-system table at all, then you'll continue to get the legacy sdist handling, allowing the addition of other tool config without impacting the way your sdist gets built.
If you do add a build-system table, then you have to populates the "requires" field properly, even if you're using setuptools as your build backend.
That way, the "build-system.backend defaults to setuptools" behaviour is only there to support pyproject.toml files that have already opted in to build isolation by writing:
[build-system] requires = ["setuptools", "wheel"]
That sounds fair to me. In terms of PEP wording: 1. build-system.requires becomes *optional* in pyproject.toml 2. Tools should process projects without pyproject.toml in the same way as they always have (backward compatibility). For pip, that means no build isolation, and the old-style processing path. 3. Tools should treat projects with pyproject.toml, but with *no* build-system.requires key the same way as (2). 4. Tools can assume that no legacy behaviour is needed for projects that specify pyproject.toml and build-system.requires. Moving forward to PEP 517, we'd allow a default for build-system.backend purely as a convenience because PEP 518 was implemented before PEP 517 - but there's no intention or commitment to retain *current* PEP 518 code paths once PEP 517 is implemented (i.e, nobody's suggesting that `build-system.backend missing` should *ever* be different from `build-system.backend = "setuptools"`). Any objections? Specifically Brett made the point that this means that as a community we're OK with pyproject.toml being the standard location for tool configuration, and not just for specifying build tools. I guess I'm personally OK with this (although I do feel that it's something we didn't fully talk through when writing the PEP, and we're now getting pushed down this route by circumstance). It might warrant a change to the PEP title, just to clarify the modified intent. Nathaniel - are you happy with this variant rather than the one you proposed? Is someone going to prepare PRs for the PEP and for pip to match this behaviour? It sounds important enough that it should probably go into pip 18.0 (although that's a discussion we can have on the pip tracker - I'm not sure there's much we need to do, and there may be a PR already). Paul PS For completeness, here's the range of options. Shout if I misrepresented anything. pyproject.toml build-system.requires Behaviour -------------- --------------------- ----------------------------- Does not exist N/A Legacy Exists Does not exist Current: Invalid file (error) Thomas: Legacy Nathaniel: Isolated (setuptools, wheel) Exists Exists Isolated (full PEP 518)
On Sat, 7 Jul 2018 at 03:42 Paul Moore <p.f.moore@gmail.com> wrote:
On 30 June 2018 at 06:33, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 June 2018 at 11:37, Nathaniel Smith <njs@pobox.com> wrote:
So my inclination is to plan on ending up with build-system.requires defaulting to ["setuptools", "wheel"], and build-system.backend defaulting to "setuptools". Hopefully we'll eventually get to a place where ~no-one uses these defaults, but carrying around the code to handle the defaults isn't really a burden.
While I was going to post to say I liked this approach, after a bit of reflection, I realised I prefer Thomas Kluyver's suggestion: instead of taking "pyproject.toml" as indicating a build-isolation compatible sdist file, instead make "pyproject.toml with a build-system table" the marker for that case.
As far as I can see, the only difference this makes is that it means pip retains the legacy (non-isolated) behaviour in a few more places (specifically places where it's quite likley the project hasn't thought about build isolation). So it's basically a slightly more forgiving version of Nathaniel's proposal.
The part of Nathaniel's approach that I think would be most confusing is a project that currently uses setup_requires which adds a pyproject.toml for (say) towncrier. The build would become isolated, but setup_requires (which is implemented by setuptools, not pip) would ignore the isolated environment and install in the wrong place (maybe? I honestly don't know). I'm quite happy to call this deprecated behaviour and point out that the project should switch to explicitly using PEP 518, but given that this whole discussion is because people haven't done that, I suspect Nathaniel's proposal doesn't actually solve the root issue here...
If you don't have a build-system table at all, then you'll continue to get the legacy sdist handling, allowing the addition of other tool config without impacting the way your sdist gets built.
If you do add a build-system table, then you have to populates the "requires" field properly, even if you're using setuptools as your build backend.
That way, the "build-system.backend defaults to setuptools" behaviour is only there to support pyproject.toml files that have already opted in to build isolation by writing:
[build-system] requires = ["setuptools", "wheel"]
That sounds fair to me. In terms of PEP wording:
1. build-system.requires becomes *optional* in pyproject.toml 2. Tools should process projects without pyproject.toml in the same way as they always have (backward compatibility). For pip, that means no build isolation, and the old-style processing path. 3. Tools should treat projects with pyproject.toml, but with *no* build-system.requires key the same way as (2). 4. Tools can assume that no legacy behaviour is needed for projects that specify pyproject.toml and build-system.requires.
Moving forward to PEP 517, we'd allow a default for build-system.backend purely as a convenience because PEP 518 was implemented before PEP 517 - but there's no intention or commitment to retain *current* PEP 518 code paths once PEP 517 is implemented (i.e, nobody's suggesting that `build-system.backend missing` should *ever* be different from `build-system.backend = "setuptools"`).
Any objections?
I do have a question about whether we specifically want the existence of build-system.requires or the existence of the entire build-system section to trigger the fallback behaviour? Is there a situation where any PEP-defined key in that section makes sense without 'requires'? IOW should having a build-system section imply the requirement of build-system.requires and leaving that key out is an error, while leaving the entire section out is now optional and is what triggers the fallback behaviour? I personally vote to say that if you define build-system you should go all-in and those it's an error to be missing 'requires' in that case.
Specifically Brett made the point that this means that as a community we're OK with pyproject.toml being the standard location for tool configuration, and not just for specifying build tools. I guess I'm personally OK with this (although I do feel that it's something we didn't fully talk through when writing the PEP, and we're now getting pushed down this route by circumstance). It might warrant a change to the PEP title, just to clarify the modified intent.
Yep, the PEP would need a quick once-over to make sure the wording makes sense if this is the final decision, new packaging BDFL. ;)
Nathaniel - are you happy with this variant rather than the one you proposed?
Is someone going to prepare PRs for the PEP and for pip to match this behaviour?
I can handle any PEP updates once we have settled and the exact change we want.
It sounds important enough that it should probably go into pip 18.0 (although that's a discussion we can have on the pip tracker - I'm not sure there's much we need to do, and there may be a PR already).
Paul
PS For completeness, here's the range of options. Shout if I misrepresented anything.
pyproject.toml build-system.requires Behaviour -------------- --------------------- ----------------------------- Does not exist N/A Legacy Exists Does not exist Current: Invalid file (error) Thomas: Legacy Nathaniel: Isolated (setuptools, wheel) Exists Exists Isolated (full PEP 518)
So I think my suggestion changes this only such that build-system.requires becomes build-system, in which case it's a Thomas/Nathaniel debate, and then having build-system leading to the current behaviour.
On 7 July 2018 at 19:56, Brett Cannon <brett@python.org> wrote:
I do have a question about whether we specifically want the existence of build-system.requires or the existence of the entire build-system section to trigger the fallback behaviour? Is there a situation where any PEP-defined key in that section makes sense without 'requires'? IOW should having a build-system section imply the requirement of build-system.requires and leaving that key out is an error, while leaving the entire section out is now optional and is what triggers the fallback behaviour?
Isolation vs legacy behaviour is a pip implementation detail and/or PEP 517 related (isolation is only mentioned in PEP 517, not in PEP 518). So PEP 518 can remain silent on this point.
I personally vote to say that if you define build-system you should go all-in and those it's an error to be missing 'requires' in that case.
This *is* a PEP 518 issue, and I'm inclined to agree with you. If Nathaniel is OK with that, I'd say that as PEP authors, you should just make that change. It's not going to make any difference to any of the real-world use cases we've seen so far, so clarifying now should be uncontroversial. Paul
On Sat, 7 Jul 2018, 16:12 Paul Moore, <p.f.moore@gmail.com> wrote:
On 30 June 2018 at 06:33, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 June 2018 at 11:37, Nathaniel Smith <njs@pobox.com> wrote:
So my inclination is to plan on ending up with build-system.requires defaulting to ["setuptools", "wheel"], and build-system.backend defaulting to "setuptools". Hopefully we'll eventually get to a place where ~no-one uses these defaults, but carrying around the code to handle the defaults isn't really a burden.
While I was going to post to say I liked this approach, after a bit of reflection, I realised I prefer Thomas Kluyver's suggestion: instead of taking "pyproject.toml" as indicating a build-isolation compatible sdist file, instead make "pyproject.toml with a build-system table" the marker for that case.
As far as I can see, the only difference this makes is that it means pip retains the legacy (non-isolated) behaviour in a few more places (specifically places where it's quite likley the project hasn't thought about build isolation). So it's basically a slightly more forgiving version of Nathaniel's proposal.
The part of Nathaniel's approach that I think would be most confusing is a project that currently uses setup_requires which adds a pyproject.toml for (say) towncrier. The build would become isolated, but setup_requires (which is implemented by setuptools, not pip) would ignore the isolated environment and install in the wrong place (maybe? I honestly don't know). I'm quite happy to call this deprecated behaviour and point out that the project should switch to explicitly using PEP 518, but given that this whole discussion is because people haven't done that, I suspect Nathaniel's proposal doesn't actually solve the root issue here...
If you don't have a build-system table at all, then you'll continue to get the legacy sdist handling, allowing the addition of other tool config without impacting the way your sdist gets built.
If you do add a build-system table, then you have to populates the "requires" field properly, even if you're using setuptools as your build backend.
That way, the "build-system.backend defaults to setuptools" behaviour is only there to support pyproject.toml files that have already opted in to build isolation by writing:
[build-system] requires = ["setuptools", "wheel"]
That sounds fair to me. In terms of PEP wording:
1. build-system.requires becomes *optional* in pyproject.toml 2. Tools should process projects without pyproject.toml in the same way as they always have (backward compatibility). For pip, that means no build isolation, and the old-style processing path. 3. Tools should treat projects with pyproject.toml, but with *no* build-system.requires key the same way as (2). 4. Tools can assume that no legacy behaviour is needed for projects that specify pyproject.toml and build-system.requires.
Moving forward to PEP 517, we'd allow a default for build-system.backend purely as a convenience because PEP 518 was implemented before PEP 517 - but there's no intention or commitment to retain *current* PEP 518 code paths once PEP 517 is implemented (i.e, nobody's suggesting that `build-system.backend missing` should *ever* be different from `build-system.backend = "setuptools"`).
Any objections? Specifically Brett made the point that this means that
as a community we're OK with pyproject.toml being the standard
location for tool configuration, and not just for specifying build tools. I guess I'm personally OK with this (although I do feel that it's something we didn't fully talk through when writing the PEP, and we're now getting pushed down this route by circumstance). It might warrant a change to the PEP title, just to clarify the modified intent.
Nathaniel - are you happy with this variant rather than the one you proposed?
Is someone going to prepare PRs for the PEP and for pip to match this behaviour? It sounds important enough that it should probably go into pip 18.0 (although that's a discussion we can have on the pip tracker - I'm not sure there's much we need to do, and there may be a PR already).
I'll be happy to update pip for this after the PEP update. It's probably gonna be updating a conditional or so. :)
Paul
PS For completeness, here's the range of options. Shout if I misrepresented anything.
pyproject.toml build-system.requires Behaviour -------------- --------------------- ----------------------------- Does not exist N/A Legacy Exists Does not exist Current: Invalid file (error) Thomas: Legacy Nathaniel: Isolated (setuptools, wheel) Exists Exists Isolated (full PEP 518)
The pip 10 behavior is not erroring out. That's why we're having this discussion. ;) It's wrapped with an unexpected change in behavior for users (isolation and everything that brings) - it's basically what Nathaniel suggested. Erroring out is what I'd proposed initially in this thread, maybe that caused this confusion. I still prefer erroring out and providing transitory escape hatches, so I'll make another push for in a follow up mail (going into a flight). Pradyun --
Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/5...
On Sat, Jul 7, 2018 at 3:41 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 June 2018 at 06:33, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 June 2018 at 11:37, Nathaniel Smith <njs@pobox.com> wrote:
So my inclination is to plan on ending up with build-system.requires defaulting to ["setuptools", "wheel"], and build-system.backend defaulting to "setuptools". Hopefully we'll eventually get to a place where ~no-one uses these defaults, but carrying around the code to handle the defaults isn't really a burden.
While I was going to post to say I liked this approach, after a bit of reflection, I realised I prefer Thomas Kluyver's suggestion: instead of taking "pyproject.toml" as indicating a build-isolation compatible sdist file, instead make "pyproject.toml with a build-system table" the marker for that case.
As far as I can see, the only difference this makes is that it means pip retains the legacy (non-isolated) behaviour in a few more places (specifically places where it's quite likley the project hasn't thought about build isolation). So it's basically a slightly more forgiving version of Nathaniel's proposal.
The part of Nathaniel's approach that I think would be most confusing is a project that currently uses setup_requires which adds a pyproject.toml for (say) towncrier. The build would become isolated, but setup_requires (which is implemented by setuptools, not pip) would ignore the isolated environment and install in the wrong place (maybe? I honestly don't know). I'm quite happy to call this deprecated behaviour and point out that the project should switch to explicitly using PEP 518, but given that this whole discussion is because people haven't done that, I suspect Nathaniel's proposal doesn't actually solve the root issue here...
Re: the interaction of build isolation and setup_requires: it looks like this is totally fine, actually. Based on some experiments + checking the docs, it appears that setup_requires has always done some magic where it doesn't actually try to install the requested packages into the current environment, but instead drops them inside the build directory and then uses some import system tricks to "overlay" them onto the current process's python path. So build isolation + setup_requires actually work very well together. I think in the long run, we want to enable build isolation everywhere. Packages that break when installed with build isolation are already broken when running 'pip install' in a fresh virtualenv. There probably are a few of these out there still that say things like "before installing this package, please install these other packages, as a separate call to pip", but it's been a long time now since I've seen one of those. And since they're already asking users to follow some finicky manual install procedure, requiring --no-build-isolation isn't a big deal. So, I don't care that much about what we use to trigger build isolation mode, because it's only a temporary thing anyway. The value of keying off something involving pyproject.toml is that it automatically gives us a kind of soft rollout: people adopting pyproject.toml are probably more willing to put up with issues with new packaging features, so we can hopefully shake out any problems before it becomes the standard. This suggests that our decision should be based on: if we want to be relatively more aggressive about rolling out build isolation, then we should key on the existence of pyproject.toml. If we want to be relatively more conservative, then we should key on the existence of build-system.requires.
If you don't have a build-system table at all, then you'll continue to get the legacy sdist handling, allowing the addition of other tool config without impacting the way your sdist gets built.
If you do add a build-system table, then you have to populates the "requires" field properly, even if you're using setuptools as your build backend.
That way, the "build-system.backend defaults to setuptools" behaviour is only there to support pyproject.toml files that have already opted in to build isolation by writing:
[build-system] requires = ["setuptools", "wheel"]
That sounds fair to me. In terms of PEP wording:
1. build-system.requires becomes *optional* in pyproject.toml 2. Tools should process projects without pyproject.toml in the same way as they always have (backward compatibility). For pip, that means no build isolation, and the old-style processing path. 3. Tools should treat projects with pyproject.toml, but with *no* build-system.requires key the same way as (2). 4. Tools can assume that no legacy behaviour is needed for projects that specify pyproject.toml and build-system.requires.
Moving forward to PEP 517, we'd allow a default for build-system.backend purely as a convenience because PEP 518 was implemented before PEP 517 - but there's no intention or commitment to retain *current* PEP 518 code paths once PEP 517 is implemented (i.e, nobody's suggesting that `build-system.backend missing` should *ever* be different from `build-system.backend = "setuptools"`).
I think that once we make build isolation the default, this proposal and my proposal will become equivalent: To use build-isolation when there's no pyproject.toml, or no build-system.requires, then obviously we'll need to default to installing ["setuptools", "wheel"] into the isolated environment. And then we'll need to run 'setup.py', like with the legacy system... which is exactly what the setuptools build-backend will do. So there won't be any reason for pip to keep carrying around its own copy of the setup.py invocation code; it can handle legacy packages through build-system.backend = "setuptools". In fact I guess that's what you just said at the end of the quoted paragraph. So basically the end state is the same either way: if build-system.requires is missing, we default to ["setuptools", "wheel"], and if build-system.backend is missing, we default to "setuptools", and that's it, no complicated legacy code-paths inside pip. So, it probably doesn't matter too much which we choose. [Hmm, I just noticed that we're both using "build-system.backend" as the name of the key, but PEP 517 actually uses "build-system.build-backend". The PEP's name is a bit redundant, isn't it.]
Any objections? Specifically Brett made the point that this means that as a community we're OK with pyproject.toml being the standard location for tool configuration, and not just for specifying build tools. I guess I'm personally OK with this (although I do feel that it's something we didn't fully talk through when writing the PEP, and we're now getting pushed down this route by circumstance). It might warrant a change to the PEP title, just to clarify the modified intent.
I'm fine with pyproject.toml becoming more central. If anything I think we might want to take this further. Maybe I'll start a thread about that :-).
Nathaniel - are you happy with this variant rather than the one you proposed?
If we're agreed that we will eventually make build-isolation the default, then I think either plan is fine. I guess I still like my proposal *slightly* better, because it seems a bit simpler to explain, and it gets us more testing of build-isolation mode, sooner. But I'm not going to make a fuss about it. If there's some reason we *don't* plan to eventually make build-isolation mode the default, and will need to keep legacy code paths inside pip indefinitely, then that makes this much more complicated and I'd want to think it through more carefully. Hopefully no-one sees any terrible flaws with making build-isolation the default (eventually)? -n -- Nathaniel J. Smith -- https://vorpus.org
On 8 July 2018 at 04:42, Nathaniel Smith <njs@pobox.com> wrote: (I'm going to read your full mail in more detail before commenting, but I wanted to pick out this point immediately, as I think it's worthwhile addressing it in particular)
If there's some reason we *don't* plan to eventually make build-isolation mode the default, and will need to keep legacy code paths inside pip indefinitely, then that makes this much more complicated and I'd want to think it through more carefully. Hopefully no-one sees any terrible flaws with making build-isolation the default (eventually)?
In principle, I'm 100% on board with this. However, as is typical with packaging changes, we have a huge backward compatibility situation to address - the many packages on PyPI that are infrequently updated or maintained and yet still in common use. We also have a large body of users who build pip into their internal workflows, who often rely heavily on what to "normal pip usage" are obscure edge cases. I'd like to find some way of assessing the impact before we simply switch to full build isolation (we've already had a fair number of bug reports on pip that are triggered by build isolation, such as the ones that started this discussion). The key question is how many users would be affected, and how much effort would it be for them to change. I'm going to make a point now that I've made before - this *only* affects people building from source, and in theory we're aiming for a world where "most" users do "pip install XXX" and download and install a set of pre-built wheels. In that world, PEP 518 behaviour is a relatively uncommon case, which is mostly needed by people who have the technical skills to deal with the change. But the pip issue tracker (which is hugely biased, as no-one raises an issue to say "I used pip and it worked - thanks!", sadly ;-)) seems to indicate that there's a lot more people building from source than we'd like to believe. So to summarise: Yes, build isolation as the default is what we want (IMO), but it's not at all clear to me how fast we can remove the legacy code paths, or how we can best smooth over the transition for our existing users. So for the moment, I'm assuming that some form of legacy (non-isolated) code path may be around for quite a while yet. Paul PS On a purely personal note, I'd prefer that we go full build isolation and drop the legacy code paths ASAP. But I'm very firmly in the "installs everything from wheels" user base, so it's not me that would suffer if we got this wrong...
On Sun, Jul 8, 2018, 04:25 Paul Moore <p.f.moore@gmail.com> wrote:
In principle, I'm 100% on board with this. However, as is typical with packaging changes, we have a huge backward compatibility situation to address - the many packages on PyPI that are infrequently updated or maintained and yet still in common use. We also have a large body of users who build pip into their internal workflows, who often rely heavily on what to "normal pip usage" are obscure edge cases. I'd like to find some way of assessing the impact before we simply switch to full build isolation (we've already had a fair number of bug reports on pip that are triggered by build isolation, such as the ones that started this discussion).
I think it would be helpful for this discussion if we could look at these bug reports – do does anyone have links? Every new rollout will cause some issues. The crucial thing IME is to see whether they're bugs in the implementation we can fix, whether there's a simple way to improve error messages for other cases, ... etc. If they're intractable issues that's very different than if they're teething pains :-) -n
On 2018-07-08 07:41:43 -0700 (-0700), Nathaniel Smith wrote:
On Sun, Jul 8, 2018, 04:25 Paul Moore <p.f.moore@gmail.com> wrote: [...]
I'd like to find some way of assessing the impact before we simply switch to full build isolation (we've already had a fair number of bug reports on pip that are triggered by build isolation, such as the ones that started this discussion).
I think it would be helpful for this discussion if we could look at these bug reports – do does anyone have links? [...]
The discussion around the attempts to resolve https://github.com/pypa/pip/issues/2195 is probably relevant, if you're looking for an example. -- Jeremy Stanley
On Sun, Jul 8, 2018, 08:32 Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2018-07-08 07:41:43 -0700 (-0700), Nathaniel Smith wrote:
On Sun, Jul 8, 2018, 04:25 Paul Moore <p.f.moore@gmail.com> wrote: [...]
I'd like to find some way of assessing the impact before we simply switch to full build isolation (we've already had a fair number of bug reports on pip that are triggered by build isolation, such as the ones that started this discussion).
I think it would be helpful for this discussion if we could look at these bug reports – do does anyone have links? [...]
The discussion around the attempts to resolve https://github.com/pypa/pip/issues/2195 is probably relevant, if you're looking for an example.
Unless I'm missing something, that's unrelated :-). The "build isolation" we're talking about here is a new feature where pip tries to use a clean python environment for the build, and that issue is about an old feature where pip tries to use a clean copy of the source directory for the build. -n
On 2018-07-08 10:31:04 -0700 (-0700), Nathaniel Smith wrote: [...]
Unless I'm missing something, that's unrelated :-). The "build isolation" we're talking about here is a new feature where pip tries to use a clean python environment for the build, and that issue is about an old feature where pip tries to use a clean copy of the source directory for the build. [...]
Ahh, yes, apologies! I had incorrectly assumed the new build environment isolation was also going to have changes to build directory isolation. -- Jeremy Stanley
On 8 July 2018 at 15:41, Nathaniel Smith <njs@pobox.com> wrote:
On Sun, Jul 8, 2018, 04:25 Paul Moore <p.f.moore@gmail.com> wrote:
In principle, I'm 100% on board with this. However, as is typical with packaging changes, we have a huge backward compatibility situation to address - the many packages on PyPI that are infrequently updated or maintained and yet still in common use. We also have a large body of users who build pip into their internal workflows, who often rely heavily on what to "normal pip usage" are obscure edge cases. I'd like to find some way of assessing the impact before we simply switch to full build isolation (we've already had a fair number of bug reports on pip that are triggered by build isolation, such as the ones that started this discussion).
I think it would be helpful for this discussion if we could look at these bug reports – do does anyone have links?
Good point. I will hunt them out and post here.
Every new rollout will cause some issues. The crucial thing IME is to see whether they're bugs in the implementation we can fix, whether there's a simple way to improve error messages for other cases, ... etc. If they're intractable issues that's very different than if they're teething pains :-)
Agreed. My impression (and it is only an impression) is that the main issue is with people being taken by surprise with new behaviour, which is a combination of things: 1. Poor communication (as noted earlier, we struggle to even reach many of our users) 2. A general principle of "you should use the latest pip" (i.e., our check and warning for if you're using the current version) 3. People using pip in ways we didn't consider I think it's important that we don't let the standardisation and improvement process get stalled by a fear of affecting people's workflows, but at the same time I think we should consider transition paths. Paul
(I thought I'd sent this but it was in my drafts instead) On Sun, Jul 8, 2018 at 9:14 AM Nathaniel Smith <njs@pobox.com> wrote:
On Sat, Jul 7, 2018 at 3:41 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 June 2018 at 06:33, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 June 2018 at 11:37, Nathaniel Smith <njs@pobox.com> wrote:
So my inclination is to plan on ending up with build-system.requires defaulting to ["setuptools", "wheel"], and build-system.backend defaulting to "setuptools". Hopefully we'll eventually get to a place where ~no-one uses these defaults, but carrying around the code to handle the defaults isn't really a burden.
While I was going to post to say I liked this approach, after a bit of reflection, I realised I prefer Thomas Kluyver's suggestion: instead of taking "pyproject.toml" as indicating a build-isolation compatible sdist file, instead make "pyproject.toml with a build-system table" the marker for that case.
As far as I can see, the only difference this makes is that it means pip retains the legacy (non-isolated) behaviour in a few more places (specifically places where it's quite likley the project hasn't thought about build isolation). So it's basically a slightly more forgiving version of Nathaniel's proposal.
The part of Nathaniel's approach that I think would be most confusing is a project that currently uses setup_requires which adds a pyproject.toml for (say) towncrier. The build would become isolated, but setup_requires (which is implemented by setuptools, not pip) would ignore the isolated environment and install in the wrong place (maybe? I honestly don't know). I'm quite happy to call this deprecated behaviour and point out that the project should switch to explicitly using PEP 518, but given that this whole discussion is because people haven't done that, I suspect Nathaniel's proposal doesn't actually solve the root issue here...
Re: the interaction of build isolation and setup_requires: it looks like this is totally fine, actually. Based on some experiments + checking the docs, it appears that setup_requires has always done some magic where it doesn't actually try to install the requested packages into the current environment, but instead drops them inside the build directory and then uses some import system tricks to "overlay" them onto the current process's python path. So build isolation + setup_requires actually work very well together.
I think in the long run, we want to enable build isolation everywhere. Packages that break when installed with build isolation are already broken when running 'pip install' in a fresh virtualenv. There probably are a few of these out there still that say things like "before installing this package, please install these other packages, as a separate call to pip", but it's been a long time now since I've seen one of those. And since they're already asking users to follow some finicky manual install procedure, requiring --no-build-isolation isn't a big deal.
So, I don't care that much about what we use to trigger build isolation mode, because it's only a temporary thing anyway. The value of keying off something involving pyproject.toml is that it automatically gives us a kind of soft rollout: people adopting pyproject.toml are probably more willing to put up with issues with new packaging features, so we can hopefully shake out any problems before it becomes the standard.
This suggests that our decision should be based on: if we want to be relatively more aggressive about rolling out build isolation, then we should key on the existence of pyproject.toml. If we want to be relatively more conservative, then we should key on the existence of build-system.requires.
If you don't have a build-system table at all, then you'll continue to get the legacy sdist handling, allowing the addition of other tool config without impacting the way your sdist gets built.
If you do add a build-system table, then you have to populates the "requires" field properly, even if you're using setuptools as your build backend.
That way, the "build-system.backend defaults to setuptools" behaviour is only there to support pyproject.toml files that have already opted in to build isolation by writing:
[build-system] requires = ["setuptools", "wheel"]
That sounds fair to me. In terms of PEP wording:
1. build-system.requires becomes *optional* in pyproject.toml 2. Tools should process projects without pyproject.toml in the same way as they always have (backward compatibility). For pip, that means no build isolation, and the old-style processing path. 3. Tools should treat projects with pyproject.toml, but with *no* build-system.requires key the same way as (2). 4. Tools can assume that no legacy behaviour is needed for projects that specify pyproject.toml and build-system.requires.
Moving forward to PEP 517, we'd allow a default for build-system.backend purely as a convenience because PEP 518 was implemented before PEP 517 - but there's no intention or commitment to retain *current* PEP 518 code paths once PEP 517 is implemented (i.e, nobody's suggesting that `build-system.backend missing` should *ever* be different from `build-system.backend = "setuptools"`).
I think that once we make build isolation the default, this proposal and my proposal will become equivalent: To use build-isolation when there's no pyproject.toml, or no build-system.requires, then obviously we'll need to default to installing ["setuptools", "wheel"] into the isolated environment. And then we'll need to run 'setup.py', like with the legacy system... which is exactly what the setuptools build-backend will do. So there won't be any reason for pip to keep carrying around its own copy of the setup.py invocation code; it can handle legacy packages through build-system.backend = "setuptools". In fact I guess that's what you just said at the end of the quoted paragraph.
So basically the end state is the same either way: if build-system.requires is missing, we default to ["setuptools", "wheel"], and if build-system.backend is missing, we default to "setuptools", and that's it, no complicated legacy code-paths inside pip.
So, it probably doesn't matter too much which we choose.
[Hmm, I just noticed that we're both using "build-system.backend" as the name of the key, but PEP 517 actually uses "build-system.build-backend". The PEP's name is a bit redundant, isn't it.]
Any objections? Specifically Brett made the point that this means that as a community we're OK with pyproject.toml being the standard location for tool configuration, and not just for specifying build tools. I guess I'm personally OK with this (although I do feel that it's something we didn't fully talk through when writing the PEP, and we're now getting pushed down this route by circumstance). It might warrant a change to the PEP title, just to clarify the modified intent.
I'm fine with pyproject.toml becoming more central. If anything I think we might want to take this further. Maybe I'll start a thread about that :-).
Nathaniel - are you happy with this variant rather than the one you
Indeed. I too feel that's what this comes down to. I had a wordier way to come to this conclusion which I've removed from this mail now. :) An important thing here is that being aggressive here let's us piggy-back on the adoption of tools that use pyproject.toml and I'd say it's a good thing to have more people using the standard explicitly. Having to specify the build-system.requires in pyproject.toml isn't going to be a problem for most (almost all?) packages. The status quo is that *they are using isolation* if there's a pyproject.toml. It won't break/change anything if they do specify the key in a future release within the pyproject.toml that's already there. In fact, requiring packages specify this information would serve to make people more familiar with it. FWIW, if build isolation is causing issues for a package, they'll have the same UX as today (passing --no-build-isolation) if we go ahead with compulsory requires. As for the existing released packages with missing requires, we'll have an escape hatch (the same --no-build-isolation or something more targeted if we want that'd be an implementation thing) that'll work just fine for such packages too; a transitory warning followed by an error message seems like a smooth enough transition to me. pip 10's PEP 518 implementation was clearly documented to be incomplete so IMO making changes isn't really a huge problem here, especially given that there's a clean-ish transition path available for nearly every use case involved (users, packagers and the entire grid of legacy/missing/specified). I feel that if a project has been quick to adopt a new standard that's documented as implemented incompletely currently, it seems reasonable to make a 2 line change to comply with the standard as it matures. The fact that the project did it for a tool doesn't exempt them from this IMO. Basically, my position is that being more aggressive is a better way to go about this than being conservative - having to a way to use the legacy behavior for users that are benefiting from our newly specified standard, whose intended use was/is to move them away from the legacy behavior, just feels weird to me. Pradyun (on mobile) proposed?
If we're agreed that we will eventually make build-isolation the default, then I think either plan is fine. I guess I still like my proposal *slightly* better, because it seems a bit simpler to explain, and it gets us more testing of build-isolation mode, sooner. But I'm not going to make a fuss about it.
If there's some reason we *don't* plan to eventually make build-isolation mode the default, and will need to keep legacy code paths inside pip indefinitely, then that makes this much more complicated and I'd want to think it through more carefully. Hopefully no-one sees any terrible flaws with making build-isolation the default (eventually)?
-n
-- Nathaniel J. Smith -- https://vorpus.org -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/U...
On 9 July 2018 at 20:27, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Sun, Jul 8, 2018 at 9:14 AM Nathaniel Smith <njs@pobox.com> wrote:
This suggests that our decision should be based on: if we want to be relatively more aggressive about rolling out build isolation, then we should key on the existence of pyproject.toml. If we want to be relatively more conservative, then we should key on the existence of build-system.requires.
Indeed. I too feel that's what this comes down to. I had a wordier way to come to this conclusion which I've removed from this mail now. :)
An important thing here is that being aggressive here let's us piggy-back on the adoption of tools that use pyproject.toml and I'd say it's a good thing to have more people using the standard explicitly.
OK. That's basically pip's current behaviour. This discussion is getting fragmented, unfortunately. I've just commented on the pip issue at https://github.com/pypa/pip/issues/5416#issuecomment-403616780 as I'm still trying to find the motivating issues behind this discussion. For now I'll point out that PEP 518 doesn't say *anything* about how tools use the information in `pyproject.toml` - there's no mention of build isolation. Unless I missed something - please point it out if I did, The only thing I can find is in PEP 517. So discussions of pip's isolation behaviour are mostly pip-specific implementation details at the moment, and not really relevant to this thread. Once I've collected a bit more information, I'll summarise here. But I think there's only minor changes to PEP 518 needed, and nothing for pip. Paul
On Tue, Jul 10, 2018 at 2:30 AM Paul Moore <p.f.moore@gmail.com> wrote:
On 9 July 2018 at 20:27, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Sun, Jul 8, 2018 at 9:14 AM Nathaniel Smith <njs@pobox.com> wrote:
This suggests that our decision should be based on: if we want to be relatively more aggressive about rolling out build isolation, then we should key on the existence of pyproject.toml. If we want to be relatively more conservative, then we should key on the existence of build-system.requires.
Indeed. I too feel that's what this comes down to. I had a wordier way to come to this conclusion which I've removed from this mail now. :)
An important thing here is that being aggressive here let's us piggy-back on the adoption of tools that use pyproject.toml and I'd say it's a good thing to have more people using the standard explicitly.
OK. That's basically pip's current behaviour.
This discussion is getting fragmented, unfortunately. I've just commented on the pip issue at https://github.com/pypa/pip/issues/5416#issuecomment-403616780 as I'm still trying to find the motivating issues behind this discussion.
For now I'll point out that PEP 518 doesn't say *anything* about how tools use the information in `pyproject.toml` - there's no mention of build isolation. Unless I missed something - please point it out if I did, The only thing I can find is in PEP 517. So discussions of pip's isolation behaviour are mostly pip-specific implementation details at the moment, and not really relevant to this thread.
Ah, okay. So, isolation is purely an implementation issue, so it doesn't need to come around in this discussion which is about how we should change the PEP. I guess I'm still figuring out where to draw the line between implementation details and the PEP details here since they should/would influence each other both ways. I'll try to be more careful about stuff like this in the future. :) Pradyun
Once I've collected a bit more information, I'll summarise here. But I think there's only minor changes to PEP 518 needed, and nothing for pip.
Paul
On 10 July 2018 at 05:09, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Tue, Jul 10, 2018 at 2:30 AM Paul Moore <p.f.moore@gmail.com> wrote:
For now I'll point out that PEP 518 doesn't say *anything* about how tools use the information in `pyproject.toml` - there's no mention of build isolation. Unless I missed something - please point it out if I did, The only thing I can find is in PEP 517. So discussions of pip's isolation behaviour are mostly pip-specific implementation details at the moment, and not really relevant to this thread.
Ah, okay. So, isolation is purely an implementation issue, so it doesn't need to come around in this discussion which is about how we should change the PEP. I guess I'm still figuring out where to draw the line between implementation details and the PEP details here since they should/would influence each other both ways. I'll try to be more careful about stuff like this in the future. :)
Not a problem. Isolation is discussed in PEP 517, which is why I was getting confused about what was related to the standard and what to the implementation. We will need to be careful to review all this once we start implementing PEP 517, but for now at least that's a level of complexity we don't need in this discussion. Paul
The discussion on this appears to have died down. As far as I can tell, the consensus is essentially: 1. It should be legal for pyproject.toml to *not* contain a [build-system] section. 2. If a [build-system] section is present, the requires key must be present. Tools should behave as follows: 1. If [build-system] is present but requires is missing, raise an error. 2. If [build-system] is missing, they can take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"] Whether tools act differently in cases 2a and 2b is tool-dependent (for pip, we would isolate in case 2b but not in case 2a) which is why the choice is left to individual tools. That makes the "Thomas/Nathaniel" debate into a tool implementation choice, and both of the options are allowable from the perspective of the PEP. Is everyone OK with this resolution? If so, will someone raise a PR for PEP 518? I can do that if no-one else can. Paul PS Following on from
I think it would be helpful for this discussion if we could look at these bug reports – do does anyone have links?
Good point. I will hunt them out and post here.
I mentioned this on the pip issue, but the only pip problem which has been raised with the current behaviour is around cases where the user disabled PyPI access and doesn't have a local copy of setuptools/wheel, which means we can't build the isolated environment. But that's a corner case that is easily resolvable, and I don't think it needs to affect pip's choice of behaviour (much less what the PEP says). On 10 July 2018 at 08:03, Paul Moore <p.f.moore@gmail.com> wrote:
On 10 July 2018 at 05:09, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Tue, Jul 10, 2018 at 2:30 AM Paul Moore <p.f.moore@gmail.com> wrote:
For now I'll point out that PEP 518 doesn't say *anything* about how tools use the information in `pyproject.toml` - there's no mention of build isolation. Unless I missed something - please point it out if I did, The only thing I can find is in PEP 517. So discussions of pip's isolation behaviour are mostly pip-specific implementation details at the moment, and not really relevant to this thread.
Ah, okay. So, isolation is purely an implementation issue, so it doesn't need to come around in this discussion which is about how we should change the PEP. I guess I'm still figuring out where to draw the line between implementation details and the PEP details here since they should/would influence each other both ways. I'll try to be more careful about stuff like this in the future. :)
Not a problem. Isolation is discussed in PEP 517, which is why I was getting confused about what was related to the standard and what to the implementation. We will need to be careful to review all this once we start implementing PEP 517, but for now at least that's a level of complexity we don't need in this discussion.
Paul
On Mon, Jul 16, 2018, at 10:22 AM, Paul Moore wrote:
2. If [build-system] is missing, they can take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"]
Whether tools act differently in cases 2a and 2b is tool-dependent (for pip, we would isolate in case 2b but not in case 2a) which is why the choice is left to individual tools. That makes the "Thomas/Nathaniel" debate into a tool implementation choice, and both of the options are allowable from the perspective of the PEP.
Is everyone OK with this resolution?
Broadly, yes. I think it might be appropriate for the PEP to make a recommendation between 2a and 2b, even if it's not a hard requirement. In practice, all other tools will probably want to do what pip does, so we don't have packages that one tool can build but another can't. I'm not going to stamp my foot about this, though. Thomas
On 16 July 2018 at 10:45, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Mon, Jul 16, 2018, at 10:22 AM, Paul Moore wrote:
2. If [build-system] is missing, they can take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"]
Whether tools act differently in cases 2a and 2b is tool-dependent (for pip, we would isolate in case 2b but not in case 2a) which is why the choice is left to individual tools. That makes the "Thomas/Nathaniel" debate into a tool implementation choice, and both of the options are allowable from the perspective of the PEP.
Is everyone OK with this resolution?
Broadly, yes. I think it might be appropriate for the PEP to make a recommendation between 2a and 2b, even if it's not a hard requirement. In practice, all other tools will probably want to do what pip does, so we don't have packages that one tool can build but another can't.
I'm not going to stamp my foot about this, though.
My inclination would be to recommend 2b. I said that before checking whether that was your proposal or Nathaniel's ;-), and it's based mostly on gut feel and "that's what pip does now and I don't see any reason to change it" though. Paul
On Mon, Jul 16, 2018, at 11:02 AM, Paul Moore wrote:
My inclination would be to recommend 2b. I said that before checking whether that was your proposal or Nathaniel's ;-), and it's based mostly on gut feel and "that's what pip does now and I don't see any reason to change it" though.
My proposal was 2a ;-). And that's still my inclination, as we've got examples of people using pyproject.toml for unrelated purposes that shouldn't affect how the package is built.
On 16 July 2018 at 11:05, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Mon, Jul 16, 2018, at 11:02 AM, Paul Moore wrote:
My inclination would be to recommend 2b. I said that before checking whether that was your proposal or Nathaniel's ;-), and it's based mostly on gut feel and "that's what pip does now and I don't see any reason to change it" though.
My proposal was 2a ;-). And that's still my inclination, as we've got examples of people using pyproject.toml for unrelated purposes that shouldn't affect how the package is built.
OK. Unless we change pip, that means that the PEP would be suggesting that tools use a behaviour different from pip. I'm not comfortable with that (which is why I left the choice unspecified) but nor has anyone convinced me that pip should change. Real world experience with 2b in pip 10 has shown very little in the way of actual issues reported by users, and we (pip) *do* want to make isolation the norm in the long run. I understand your theoretical point, though I don't think it beats the practical considerations. Paul
On Mon, Jul 16, 2018, at 11:50 AM, Paul Moore wrote:
On 16 July 2018 at 11:05, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
My proposal was 2a ;-). And that's still my inclination, as we've got examples of people using pyproject.toml for unrelated purposes that shouldn't affect how the package is built.
OK. Unless we change pip, that means that the PEP would be suggesting that tools use a behaviour different from pip. I'm not comfortable with that (which is why I left the choice unspecified) but nor has anyone convinced me that pip should change. Real world experience with 2b in pip 10 has shown very little in the way of actual issues reported by users, and we (pip) *do* want to make isolation the norm in the long run.
IIRC pip can't do effective isolation yet, so any problems that do arise from this are still largely theoretical. I *am* suggesting that pip should switch to 2a while this functionality is still new enough to make that relatively feasible. I'm not volunteering to make the changes, though, so feel free to ignore me. ;-) Thomas
I will update the PEP and add you as a reviewer, Paul (might not get to it until Friday, though). On Mon, Jul 16, 2018, 02:23 Paul Moore, <p.f.moore@gmail.com> wrote:
The discussion on this appears to have died down.
As far as I can tell, the consensus is essentially:
1. It should be legal for pyproject.toml to *not* contain a [build-system] section. 2. If a [build-system] section is present, the requires key must be present.
Tools should behave as follows:
1. If [build-system] is present but requires is missing, raise an error. 2. If [build-system] is missing, they can take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"]
Whether tools act differently in cases 2a and 2b is tool-dependent (for pip, we would isolate in case 2b but not in case 2a) which is why the choice is left to individual tools. That makes the "Thomas/Nathaniel" debate into a tool implementation choice, and both of the options are allowable from the perspective of the PEP.
Is everyone OK with this resolution? If so, will someone raise a PR for PEP 518? I can do that if no-one else can.
Paul
PS Following on from
I think it would be helpful for this discussion if we could look at these bug reports – do does anyone have links?
Good point. I will hunt them out and post here.
I mentioned this on the pip issue, but the only pip problem which has been raised with the current behaviour is around cases where the user disabled PyPI access and doesn't have a local copy of setuptools/wheel, which means we can't build the isolated environment. But that's a corner case that is easily resolvable, and I don't think it needs to affect pip's choice of behaviour (much less what the PEP says).
On 10 July 2018 at 08:03, Paul Moore <p.f.moore@gmail.com> wrote:
On 10 July 2018 at 05:09, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Tue, Jul 10, 2018 at 2:30 AM Paul Moore <p.f.moore@gmail.com> wrote:
For now I'll point out that PEP 518 doesn't say *anything* about how tools use the information in `pyproject.toml` - there's no mention of build isolation. Unless I missed something - please point it out if I did, The only thing I can find is in PEP 517. So discussions of pip's isolation behaviour are mostly pip-specific implementation details at the moment, and not really relevant to this thread.
Ah, okay. So, isolation is purely an implementation issue, so it doesn't need to come around in this discussion which is about how we should change the PEP. I guess I'm still figuring out where to draw the line between implementation details and the PEP details here since they should/would influence each other both ways. I'll try to be more careful about stuff like this in the future. :)
Not a problem. Isolation is discussed in PEP 517, which is why I was getting confused about what was related to the standard and what to the implementation. We will need to be careful to review all this once we start implementing PEP 517, but for now at least that's a level of complexity we don't need in this discussion.
Paul
On Mon, 16 Jul 2018, 20:16 Brett Cannon, <brett@python.org> wrote:
I will update the PEP and add you as a reviewer, Paul (might not get to it until Friday, though).
On Mon, Jul 16, 2018, 02:23 Paul Moore, <p.f.moore@gmail.com> wrote:
The discussion on this appears to have died down.
As far as I can tell, the consensus is essentially:
1. It should be legal for pyproject.toml to *not* contain a [build-system] section. 2. If a [build-system] section is present, the requires key must be present.
Tools should behave as follows:
1. If [build-system] is present but requires is missing, raise an error. 2. If [build-system] is missing, they can take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"]
Whether tools act differently in cases 2a and 2b is tool-dependent (for pip, we would isolate in case 2b but not in case 2a) which is why the choice is left to individual tools. That makes the "Thomas/Nathaniel" debate into a tool implementation choice, and both of the options are allowable from the perspective of the PEP.
Is everyone OK with this resolution? If so, will someone raise a PR for PEP 518? I can do that if no-one else can.
While I don't mind if we'd go ahead with this (and 2b, to not change existing behavior), I still think making the key mandatory would be a good/better idea. The visibility (and hence familiarity) that this would gain from the specification of build-system.requires being mandatory would be nice to have. I feel that this would otherwise be a missed opportunity to push for the key and standard to be more visible to users and packagers. The transitory cost for existing packages that break due to this being mandatory would probably be fairly minor (I realize that this point is where I dwelled into implementation details earlier, so I'll not digress there now). Pradyun
Paul
PS Following on from
I think it would be helpful for this discussion if we could look at these bug reports – do does anyone have links?
Good point. I will hunt them out and post here.
I mentioned this on the pip issue, but the only pip problem which has been raised with the current behaviour is around cases where the user disabled PyPI access and doesn't have a local copy of setuptools/wheel, which means we can't build the isolated environment. But that's a corner case that is easily resolvable, and I don't think it needs to affect pip's choice of behaviour (much less what the PEP says).
On 10 July 2018 at 08:03, Paul Moore <p.f.moore@gmail.com> wrote:
On 10 July 2018 at 05:09, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Tue, Jul 10, 2018 at 2:30 AM Paul Moore <p.f.moore@gmail.com> wrote:
For now I'll point out that PEP 518 doesn't say *anything* about how tools use the information in `pyproject.toml` - there's no mention of build isolation. Unless I missed something - please point it out if I did, The only thing I can find is in PEP 517. So discussions of pip's isolation behaviour are mostly pip-specific implementation details at the moment, and not really relevant to this thread.
Ah, okay. So, isolation is purely an implementation issue, so it doesn't need to come around in this discussion which is about how we should change the PEP. I guess I'm still figuring out where to draw the line between implementation details and the PEP details here since they should/would influence each other both ways. I'll try to be more careful about stuff like this in the future. :)
Not a problem. Isolation is discussed in PEP 517, which is why I was getting confused about what was related to the standard and what to the implementation. We will need to be careful to review all this once we start implementing PEP 517, but for now at least that's a level of complexity we don't need in this discussion.
Paul
On 16 July 2018 at 15:56, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Mon, 16 Jul 2018, 20:16 Brett Cannon, <brett@python.org> wrote:
I will update the PEP and add you as a reviewer, Paul (might not get to it until Friday, though).
On Mon, Jul 16, 2018, 02:23 Paul Moore, <p.f.moore@gmail.com> wrote:
The discussion on this appears to have died down.
As far as I can tell, the consensus is essentially:
1. It should be legal for pyproject.toml to *not* contain a [build-system] section. 2. If a [build-system] section is present, the requires key must be present.
Tools should behave as follows:
1. If [build-system] is present but requires is missing, raise an error. 2. If [build-system] is missing, they can take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"]
Whether tools act differently in cases 2a and 2b is tool-dependent (for pip, we would isolate in case 2b but not in case 2a) which is why the choice is left to individual tools. That makes the "Thomas/Nathaniel" debate into a tool implementation choice, and both of the options are allowable from the perspective of the PEP.
Is everyone OK with this resolution? If so, will someone raise a PR for PEP 518? I can do that if no-one else can.
While I don't mind if we'd go ahead with this (and 2b, to not change existing behavior), I still think making the key mandatory would be a good/better idea.
I get the feeling that no-one is sufficiently motivated to argue strongly for any particular resolution - which is why my summary is basically "accept the current behaviour as correct" :-)
The visibility (and hence familiarity) that this would gain from the specification of build-system.requires being mandatory would be nice to have. I feel that this would otherwise be a missed opportunity to push for the key and standard to be more visible to users and packagers.
But adoption of pyproject.toml as a standard configuration file by projects like towncrier and black is doing that already.
The transitory cost for existing packages that break due to this being mandatory would probably be fairly minor (I realize that this point is where I dwelled into implementation details earlier, so I'll not digress there now).
I believe the transition costs for projects (and their users) using pyproject.toml for config only are: Optional, only isolate if [build-system] is present: None Optional, isolate if pyproject.toml is present: The few *end users* for whom isolation breaks the build have to pass --no-build-isolation Mandatory: The *project* has to add the [build-system] section, and make a new release. End users have to wait for that release. If I've got that right, the costs for making the key mandatory are a lot higher. The costs for 2b are there, but a lot smaller (and the end user has a workaround). Option 2a has no costs, but also misses the opportunity to move towards build isolation by default for a few extra cases. Paul
On Mon, 16 Jul 2018, 21:11 Paul Moore, <p.f.moore@gmail.com> wrote:
On 16 July 2018 at 15:56, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Mon, 16 Jul 2018, 20:16 Brett Cannon, <brett@python.org> wrote:
I will update the PEP and add you as a reviewer, Paul (might not get to
it
until Friday, though).
On Mon, Jul 16, 2018, 02:23 Paul Moore, <p.f.moore@gmail.com> wrote:
The discussion on this appears to have died down.
As far as I can tell, the consensus is essentially:
1. It should be legal for pyproject.toml to *not* contain a [build-system] section. 2. If a [build-system] section is present, the requires key must be present.
Tools should behave as follows:
1. If [build-system] is present but requires is missing, raise an
error.
2. If [build-system] is missing, they can take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"]
Whether tools act differently in cases 2a and 2b is tool-dependent (for pip, we would isolate in case 2b but not in case 2a) which is why the choice is left to individual tools. That makes the "Thomas/Nathaniel" debate into a tool implementation choice, and both of the options are allowable from the perspective of the PEP.
Is everyone OK with this resolution? If so, will someone raise a PR for PEP 518? I can do that if no-one else can.
While I don't mind if we'd go ahead with this (and 2b, to not change existing behavior), I still think making the key mandatory would be a good/better idea.
I get the feeling that no-one is sufficiently motivated to argue strongly for any particular resolution - which is why my summary is basically "accept the current behaviour as correct" :-)
I just perceive this as a case of - there's a nicer and arguably, cleaner option and there's not much cost to getting there. Let's do that. :) I don't mind the status quo though.
The visibility (and hence familiarity) that this would gain from the specification of build-system.requires being mandatory would be nice to have. I feel that this would otherwise be a missed opportunity to push for the key and standard to be more visible to users and packagers.
But adoption of pyproject.toml as a standard configuration file by projects like towncrier and black is doing that already.
I'm referring to familiarity with the build-system table (through build-system.requires being mandatory). If you have to add that table/key when using those tools, people who work on those projects get to know about the table.
The transitory cost for existing packages that break due to this being mandatory would probably be fairly minor (I realize that this point is where I dwelled into implementation details earlier, so I'll not digress there now).
I believe the transition costs for projects (and their users) using pyproject.toml for config only are:
Optional, only isolate if [build-system] is present: None Optional, isolate if pyproject.toml is present: The few *end users* for whom isolation breaks the build have to pass --no-build-isolation Mandatory: The *project* has to add the [build-system] section, and make a new release. End users have to wait for that release.
Not really. With mandatory, you can use --no-isolation - it would still work fine for basically every user that has setuptools and wheel installed in their current environment (that's almost every environment I'd say).
If I've got that right, the costs for making the key mandatory are a lot higher.
My intuition/feeling here is that this cost is worth the network effects that would make more users familiar with them. Pradyun The costs for 2b are there, but a lot smaller (and the end
user has a workaround). Option 2a has no costs, but also misses the opportunity to move towards build isolation by default for a few extra cases.
Paul
On Mon, 16 Jul 2018 at 10:47 Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Mon, 16 Jul 2018, 21:11 Paul Moore, <p.f.moore@gmail.com> wrote:
On 16 July 2018 at 15:56, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Mon, 16 Jul 2018, 20:16 Brett Cannon, <brett@python.org> wrote:
I will update the PEP and add you as a reviewer, Paul (might not get
to it
until Friday, though).
On Mon, Jul 16, 2018, 02:23 Paul Moore, <p.f.moore@gmail.com> wrote:
The discussion on this appears to have died down.
As far as I can tell, the consensus is essentially:
1. It should be legal for pyproject.toml to *not* contain a [build-system] section. 2. If a [build-system] section is present, the requires key must be present.
Tools should behave as follows:
1. If [build-system] is present but requires is missing, raise an
error.
2. If [build-system] is missing, they can take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"]
Whether tools act differently in cases 2a and 2b is tool-dependent (for pip, we would isolate in case 2b but not in case 2a) which is why the choice is left to individual tools. That makes the "Thomas/Nathaniel" debate into a tool implementation choice, and both of the options are allowable from the perspective of the PEP.
Is everyone OK with this resolution? If so, will someone raise a PR for PEP 518? I can do that if no-one else can.
While I don't mind if we'd go ahead with this (and 2b, to not change existing behavior), I still think making the key mandatory would be a good/better idea.
I get the feeling that no-one is sufficiently motivated to argue strongly for any particular resolution - which is why my summary is basically "accept the current behaviour as correct" :-)
I just perceive this as a case of - there's a nicer and arguably, cleaner option and there's not much cost to getting there. Let's do that. :)
I don't mind the status quo though.
The visibility (and hence familiarity) that this would gain from the specification of build-system.requires being mandatory would be nice to have. I feel that this would otherwise be a missed opportunity to push for the key and standard to be more visible to users and packagers.
But adoption of pyproject.toml as a standard configuration file by projects like towncrier and black is doing that already.
I'm referring to familiarity with the build-system table (through build-system.requires being mandatory). If you have to add that table/key when using those tools, people who work on those projects get to know about the table.
The transitory cost for existing packages that break due to this being mandatory would probably be fairly minor (I realize that this point is where I dwelled into implementation details earlier, so I'll not digress there now).
I believe the transition costs for projects (and their users) using pyproject.toml for config only are:
Optional, only isolate if [build-system] is present: None Optional, isolate if pyproject.toml is present: The few *end users* for whom isolation breaks the build have to pass --no-build-isolation Mandatory: The *project* has to add the [build-system] section, and make a new release. End users have to wait for that release.
Not really. With mandatory, you can use --no-isolation - it would still work fine for basically every user that has setuptools and wheel installed in their current environment (that's almost every environment I'd say).
If I've got that right, the costs for making the key mandatory are a lot higher.
My intuition/feeling here is that this cost is worth the network effects that would make more users familiar with them.
I don't think all the pieces are quite there yet. E.g. there is no `pip sdist` to go with `pip wheel`, so we still have some work to do before we can really push people to entirely drop the old way of do things with minimal penalty. -Brett
Pradyun
The costs for 2b are there, but a lot smaller (and the end
user has a workaround). Option 2a has no costs, but also misses the opportunity to move towards build isolation by default for a few extra cases.
Paul
On Tue, 17 Jul 2018, 00:45 Brett Cannon, <brett@python.org> wrote:
On Mon, 16 Jul 2018 at 10:47 Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Mon, 16 Jul 2018, 21:11 Paul Moore, <p.f.moore@gmail.com> wrote:
On 16 July 2018 at 15:56, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Mon, 16 Jul 2018, 20:16 Brett Cannon, <brett@python.org> wrote:
I will update the PEP and add you as a reviewer, Paul (might not get
to it
until Friday, though).
On Mon, Jul 16, 2018, 02:23 Paul Moore, <p.f.moore@gmail.com> wrote:
The discussion on this appears to have died down.
As far as I can tell, the consensus is essentially:
1. It should be legal for pyproject.toml to *not* contain a [build-system] section. 2. If a [build-system] section is present, the requires key must be present.
Tools should behave as follows:
1. If [build-system] is present but requires is missing, raise an
error.
2. If [build-system] is missing, they can take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"]
Whether tools act differently in cases 2a and 2b is tool-dependent (for pip, we would isolate in case 2b but not in case 2a) which is why the choice is left to individual tools. That makes the "Thomas/Nathaniel" debate into a tool implementation choice, and both of the options are allowable from the perspective of the PEP.
Is everyone OK with this resolution? If so, will someone raise a PR for PEP 518? I can do that if no-one else can.
While I don't mind if we'd go ahead with this (and 2b, to not change existing behavior), I still think making the key mandatory would be a good/better idea.
I get the feeling that no-one is sufficiently motivated to argue strongly for any particular resolution - which is why my summary is basically "accept the current behaviour as correct" :-)
I just perceive this as a case of - there's a nicer and arguably, cleaner option and there's not much cost to getting there. Let's do that. :)
I don't mind the status quo though.
The visibility (and hence familiarity) that this would gain from the specification of build-system.requires being mandatory would be nice to have. I feel that this would otherwise be a missed opportunity to push for the key and standard to be more visible to users and packagers.
But adoption of pyproject.toml as a standard configuration file by projects like towncrier and black is doing that already.
I'm referring to familiarity with the build-system table (through build-system.requires being mandatory). If you have to add that table/key when using those tools, people who work on those projects get to know about the table.
The transitory cost for existing packages that break due to this being mandatory would probably be fairly minor (I realize that this point is where I dwelled into implementation details earlier, so I'll not digress there now).
I believe the transition costs for projects (and their users) using pyproject.toml for config only are:
Optional, only isolate if [build-system] is present: None Optional, isolate if pyproject.toml is present: The few *end users* for whom isolation breaks the build have to pass --no-build-isolation Mandatory: The *project* has to add the [build-system] section, and make a new release. End users have to wait for that release.
Not really. With mandatory, you can use --no-isolation - it would still work fine for basically every user that has setuptools and wheel installed in their current environment (that's almost every environment I'd say).
If I've got that right, the costs for making the key mandatory are a lot higher.
My intuition/feeling here is that this cost is worth the network effects that would make more users familiar with them.
I don't think all the pieces are quite there yet. E.g. there is no `pip sdist` to go with `pip wheel`, so we still have some work to do before we can really push people to entirely drop the old way of do things with minimal penalty.
I do agree that not having all pieces is a problem for users currently (early adopters) and we should try to get to the point where you can entirely drop the "old way" sooner than later. :) I feel like you're making a point here that I'm not able to understand what for. This is still going to hold true with status quo until we do that work so this doesn't count against mandatory requires per se. The additional visibility of this won't be a strong push IMO until this does become a feasible alternative. Pradyun
-Brett
Pradyun
The costs for 2b are there, but a lot smaller (and the end
user has a workaround). Option 2a has no costs, but also misses the opportunity to move towards build isolation by default for a few extra cases.
Paul
My point is I don't think enough pieces are in place to force everyone over to --no-isolation to get by if they don't set up pyproject.toml as you're suggesting they be required to. IOW I think you're advocating something that's going to alienate people by pushing too hard. On Mon, Jul 16, 2018, 19:30 Pradyun Gedam, <pradyunsg@gmail.com> wrote:
On Tue, 17 Jul 2018, 00:45 Brett Cannon, <brett@python.org> wrote:
On Mon, 16 Jul 2018 at 10:47 Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Mon, 16 Jul 2018, 21:11 Paul Moore, <p.f.moore@gmail.com> wrote:
On 16 July 2018 at 15:56, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Mon, 16 Jul 2018, 20:16 Brett Cannon, <brett@python.org> wrote:
I will update the PEP and add you as a reviewer, Paul (might not get
to it
until Friday, though).
On Mon, Jul 16, 2018, 02:23 Paul Moore, <p.f.moore@gmail.com> wrote: > > The discussion on this appears to have died down. > > As far as I can tell, the consensus is essentially: > > 1. It should be legal for pyproject.toml to *not* contain a > [build-system] section. > 2. If a [build-system] section is present, the requires key must be > present. > > Tools should behave as follows: > > 1. If [build-system] is present but requires is missing, raise an error. > 2. If [build-system] is missing, they can take one of the following > two approaches: > a) Act as if pyproject.toml is missing altogether > b) Act as if [build-system] is present, with a requires value of > ["setuptools", "wheel"] > > Whether tools act differently in cases 2a and 2b is tool-dependent > (for pip, we would isolate in case 2b but not in case 2a) which is why > the choice is left to individual tools. That makes the > "Thomas/Nathaniel" debate into a tool implementation choice, and both > of the options are allowable from the perspective of the PEP. > > Is everyone OK with this resolution? If so, will someone raise a PR > for PEP 518? I can do that if no-one else can.
While I don't mind if we'd go ahead with this (and 2b, to not change existing behavior), I still think making the key mandatory would be a good/better idea.
I get the feeling that no-one is sufficiently motivated to argue strongly for any particular resolution - which is why my summary is basically "accept the current behaviour as correct" :-)
I just perceive this as a case of - there's a nicer and arguably, cleaner option and there's not much cost to getting there. Let's do that. :)
I don't mind the status quo though.
The visibility (and hence familiarity) that this would gain from the specification of build-system.requires being mandatory would be nice to have. I feel that this would otherwise be a missed opportunity to push for the key and standard to be more visible to users and packagers.
But adoption of pyproject.toml as a standard configuration file by projects like towncrier and black is doing that already.
I'm referring to familiarity with the build-system table (through build-system.requires being mandatory). If you have to add that table/key when using those tools, people who work on those projects get to know about the table.
The transitory cost for existing packages that break due to this being mandatory would probably be fairly minor (I realize that this point is where I dwelled into implementation details earlier, so I'll not digress there now).
I believe the transition costs for projects (and their users) using pyproject.toml for config only are:
Optional, only isolate if [build-system] is present: None Optional, isolate if pyproject.toml is present: The few *end users* for whom isolation breaks the build have to pass --no-build-isolation Mandatory: The *project* has to add the [build-system] section, and make a new release. End users have to wait for that release.
Not really. With mandatory, you can use --no-isolation - it would still work fine for basically every user that has setuptools and wheel installed in their current environment (that's almost every environment I'd say).
If I've got that right, the costs for making the key mandatory are a lot higher.
My intuition/feeling here is that this cost is worth the network effects that would make more users familiar with them.
I don't think all the pieces are quite there yet. E.g. there is no `pip sdist` to go with `pip wheel`, so we still have some work to do before we can really push people to entirely drop the old way of do things with minimal penalty.
I do agree that not having all pieces is a problem for users currently (early adopters) and we should try to get to the point where you can entirely drop the "old way" sooner than later. :)
I feel like you're making a point here that I'm not able to understand what for. This is still going to hold true with status quo until we do that work so this doesn't count against mandatory requires per se. The additional visibility of this won't be a strong push IMO until this does become a feasible alternative.
Pradyun
-Brett
Pradyun
The costs for 2b are there, but a lot smaller (and the end
user has a workaround). Option 2a has no costs, but also misses the opportunity to move towards build isolation by default for a few extra cases.
Paul
On Tue, 17 Jul 2018, 09:11 Brett Cannon, <brett@python.org> wrote:
My point is I don't think enough pieces are in place to force everyone over to --no-isolation to get by if they don't set up pyproject.toml as you're suggesting they be required to. IOW I think you're advocating something that's going to alienate people by pushing too hard.
Ah. I now understand what you meant. I don't think the transition costs are very high here (more on this below). I also think most projects who've adopted pyproject.toml won't mind adding 2 lines to it, given enough time. (pip-specific deprecation details follow; I'm not sure how to explain the transition related costs in any other way) Since this would be a behavior change in pip, this would have to go through the standard deprecation cycle which would probably be about as long as the feature has been available currently. This means projects would have time to make releases with the key, that's similar to the amount of time the feature has been available as a part of pip. A functional difference here is that older releases with such pyproject.toml files would need an escape hatch after the change is made, to be installable. I don't think this is a major issue since when we'd flip the switch to start refusing, we'd start suggesting users to use the escape hatch or upgrade to a newer version, since the deprecation would make projects add the key and make newer releases. And, --no-isolation can serve as the escape hatch too. Pradyun
On Mon, Jul 16, 2018, 19:30 Pradyun Gedam, <pradyunsg@gmail.com> wrote:
On Tue, 17 Jul 2018, 00:45 Brett Cannon, <brett@python.org> wrote:
On Mon, 16 Jul 2018 at 10:47 Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Mon, 16 Jul 2018, 21:11 Paul Moore, <p.f.moore@gmail.com> wrote:
On 16 July 2018 at 15:56, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Mon, 16 Jul 2018, 20:16 Brett Cannon, <brett@python.org> wrote: > > I will update the PEP and add you as a reviewer, Paul (might not
get to it
> until Friday, though). > > On Mon, Jul 16, 2018, 02:23 Paul Moore, <p.f.moore@gmail.com> wrote: >> >> The discussion on this appears to have died down. >> >> As far as I can tell, the consensus is essentially: >> >> 1. It should be legal for pyproject.toml to *not* contain a >> [build-system] section. >> 2. If a [build-system] section is present, the requires key must be >> present. >> >> Tools should behave as follows: >> >> 1. If [build-system] is present but requires is missing, raise an error. >> 2. If [build-system] is missing, they can take one of the following >> two approaches: >> a) Act as if pyproject.toml is missing altogether >> b) Act as if [build-system] is present, with a requires value of >> ["setuptools", "wheel"] >> >> Whether tools act differently in cases 2a and 2b is tool-dependent >> (for pip, we would isolate in case 2b but not in case 2a) which is why >> the choice is left to individual tools. That makes the >> "Thomas/Nathaniel" debate into a tool implementation choice, and both >> of the options are allowable from the perspective of the PEP. >> >> Is everyone OK with this resolution? If so, will someone raise a PR >> for PEP 518? I can do that if no-one else can.
While I don't mind if we'd go ahead with this (and 2b, to not change existing behavior), I still think making the key mandatory would be a good/better idea.
I get the feeling that no-one is sufficiently motivated to argue strongly for any particular resolution - which is why my summary is basically "accept the current behaviour as correct" :-)
I just perceive this as a case of - there's a nicer and arguably, cleaner option and there's not much cost to getting there. Let's do that. :)
I don't mind the status quo though.
The visibility (and hence familiarity) that this would gain from the specification of build-system.requires being mandatory would be nice to have. I feel that this would otherwise be a missed opportunity to push for the key and standard to be more visible to users and packagers.
But adoption of pyproject.toml as a standard configuration file by projects like towncrier and black is doing that already.
I'm referring to familiarity with the build-system table (through build-system.requires being mandatory). If you have to add that table/key when using those tools, people who work on those projects get to know about the table.
The transitory cost for existing packages that break due to this being mandatory would probably be fairly minor (I realize that this point is where I dwelled into implementation details earlier, so I'll not digress there now).
I believe the transition costs for projects (and their users) using pyproject.toml for config only are:
Optional, only isolate if [build-system] is present: None Optional, isolate if pyproject.toml is present: The few *end users* for whom isolation breaks the build have to pass --no-build-isolation Mandatory: The *project* has to add the [build-system] section, and make a new release. End users have to wait for that release.
Not really. With mandatory, you can use --no-isolation - it would still work fine for basically every user that has setuptools and wheel installed in their current environment (that's almost every environment I'd say).
If I've got that right, the costs for making the key mandatory are a lot higher.
My intuition/feeling here is that this cost is worth the network effects that would make more users familiar with them.
I don't think all the pieces are quite there yet. E.g. there is no `pip sdist` to go with `pip wheel`, so we still have some work to do before we can really push people to entirely drop the old way of do things with minimal penalty.
I do agree that not having all pieces is a problem for users currently (early adopters) and we should try to get to the point where you can entirely drop the "old way" sooner than later. :)
I feel like you're making a point here that I'm not able to understand what for. This is still going to hold true with status quo until we do that work so this doesn't count against mandatory requires per se. The additional visibility of this won't be a strong push IMO until this does become a feasible alternative.
Pradyun
-Brett
Pradyun
The costs for 2b are there, but a lot smaller (and the end
user has a workaround). Option 2a has no costs, but also misses the opportunity to move towards build isolation by default for a few extra cases.
Paul
On 17 July 2018 at 08:45, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Tue, 17 Jul 2018, 09:11 Brett Cannon, <brett@python.org> wrote:
My point is I don't think enough pieces are in place to force everyone over to --no-isolation to get by if they don't set up pyproject.toml as you're suggesting they be required to. IOW I think you're advocating something that's going to alienate people by pushing too hard.
Ah. I now understand what you meant.
I don't think the transition costs are very high here (more on this below). I also think most projects who've adopted pyproject.toml won't mind adding 2 lines to it, given enough time.
OK. So if we ignore pip details for the moment (I'll come back to them later) then you want to argue that 1. Projects which have currently added a pyproject.toml file with tool config but no [build-system] section are wrong to do so. At best they've misinterpreted the current PEP wording, and we're going to issue a revised PEP wording that makes it clear that they are wrong. At worst they never read the new PEP, just heard about pyproject.toml from the towncrier/black docs (and those don't mention [build-system] - see https://github.com/hawkowl/towncrier/blob/master/README.rst for example). 2. Those projects need to modify pyproject.toml, and until they do they will be in violation of the PEP (at least of the revised PEP). 2a. Towncrier need to modify their docs to *not* recommend creating an invalid pyproject.toml. 3. The only indication they get that they are in violation will come from their users using pip 18.0 and later, who will report deprecation warnings (and later failures) to them. 4. The users can work around the issue by using --no-build-isolation, but a final solution has to come from the project. 5. The projects can relatively trivially fix the issue by adding a [build-system] section to pyproject.toml, but they will need to make a new release. The costs are fairly clear here - projects that are in this situation (and we don't know how many there are, but we know there are some from pip bug reports) will need to make a new release to conform to the (revised) PEP. The route for the project to find out that there is an issue is user gets a warning from pip -> user reports the issue (probably initially to pip, and we'll have to redirect them to the project) -> project fixes the issue. There's also a PR cost, in that projects who have enthusiastically adopted pyproject,toml as being a nice "common configuration" location, will be left feeling that maybe that wasn't such a good decision if it's causing them problems like this. Plus, projects like towncrier will need to update their docs (yes, that's a bit ideal world - the reality is that people simply keep doing what they do now, and we get the message out via a gradual process of addressing bug reports and providing the explanation there). The benefits are less obvious - a slightly simpler PEP, and what? It's not as if pip gets to test out its isolation code on any additional projects, as option 2b in my proposal does that anyway. It's not that pyproject.toml gets any more visibility - projects hitting this are already using it.
(pip-specific deprecation details follow; I'm not sure how to explain the transition related costs in any other way)
See above. The *costs* are simple - projects must change and release a new version, possibly some bad PR for the pyproject.toml format. The pip-specific side is about *benefits* - are there any that justify these costs?
Since this would be a behavior change in pip, this would have to go through the standard deprecation cycle which would probably be about as long as the feature has been available currently. This means projects would have time to make releases with the key, that's similar to the amount of time the feature has been available as a part of pip.
Agreed. So starting with pip 18.0, end users would see warnings "pyproject.toml has no [build-system] section, please correct that file". But the install would proceed. We'd have to hope that those users would take the time to report that warning (as in many cases they won't be the project owners).
A functional difference here is that older releases with such pyproject.toml files would need an escape hatch after the change is made, to be installable. I don't think this is a major issue since when we'd flip the switch to start refusing, we'd start suggesting users to use the escape hatch or upgrade to a newer version, since the deprecation would make projects add the key and make newer releases. And, --no-isolation can serve as the escape hatch too.
Surely not? The point of having a deprecation warning is precisely that, after we flip the switch, we *stop* allowing the old format? This sounds like the worst of both worlds. End users can always use --no-build-isolation (at least until we finally kill off the legacy code paths altogether). But that's not a fix for having an invalid pyproject.toml, that's opting out of all of the new processing/features completely. To be frank, I actually think that tools having picked up on pyproject.toml is a good thing, as it's added publicity for PEP 518. I think we should capitalise on that and make sure that we don't break projects that have taken this step. I don't have a problem with isolating in this case, as that seems to cause little problem (and where it does, the explanation isn't hard) and it makes sure we get real-world feedback on how projects that *haven't* bought into the full PEP 517/518 workflow will be impacted. Paul
On Jul 17, 2018, at 5:27 AM, Paul Moore <p.f.moore@gmail.com> wrote:
There's also a PR cost, in that projects who have enthusiastically adopted pyproject,toml as being a nice "common configuration" location, will be left feeling that maybe that wasn't such a good decision if it's causing them problems like this. Plus, projects like towncrier will need to update their docs (yes, that's a bit ideal world - the reality is that people simply keep doing what they do now, and we get the message out via a gradual process of addressing bug reports and providing the explanation there).
I think that this is the most important reason not to go too crazy here. We should think of breaking changes as having a limited budget for introducing breaking changes, and the question then becomes where do we want to spend our budget? While something being relatively new means that the cost to our budget is smaller, it still has a cost and I just don’t think this is a great place to spend our budget at. On top of that, I don’t think the end result is significantly or meaningfully better for the end users, particularly if we ever get to a world where we isolate everything by default (which would mean that we would have to have an implicit requires of setuptools/wheel anyways).
On Tue, 17 Jul 2018, 20:44 Donald Stufft, <donald@stufft.io> wrote:
On Jul 17, 2018, at 5:27 AM, Paul Moore <p.f.moore@gmail.com> wrote:
There's also a PR cost, in that projects who have enthusiastically adopted pyproject,toml as being a nice "common configuration" location, will be left feeling that maybe that wasn't such a good decision if it's causing them problems like this. Plus, projects like towncrier will need to update their docs (yes, that's a bit ideal world - the reality is that people simply keep doing what they do now, and we get the message out via a gradual process of addressing bug reports and providing the explanation there).
I think that this is the most important reason not to go too crazy here. We should think of breaking changes as having a limited budget for introducing breaking changes, and the question then becomes where do we want to spend our budget? While something being relatively new means that the cost to our budget is smaller, it still has a cost and I just don’t think this is a great place to spend our budget at.
On top of that, I don’t think the end result is significantly or meaningfully better for the end users, particularly if we ever get to a world where we isolate everything by default (which would mean that we would have to have an implicit requires of setuptools/wheel anyways).
Fair enough. I just wanted to make the build-system table to be more visible to end users, since I feel the cost here is justified by the benefit s . If others don't agree, that's cool. I don't feel super strongly about this tbh . I am not opposed to keeping the status quo on this matter and am on board with Paul's suggestion of making the build-system table optional. :) Pradyun
On 19 July 2018 at 07:44, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Tue, 17 Jul 2018, 20:44 Donald Stufft, <donald@stufft.io> wrote:
On Jul 17, 2018, at 5:27 AM, Paul Moore <p.f.moore@gmail.com> wrote:
There's also a PR cost, in that projects who have enthusiastically adopted pyproject,toml as being a nice "common configuration" location, will be left feeling that maybe that wasn't such a good decision if it's causing them problems like this. Plus, projects like towncrier will need to update their docs (yes, that's a bit ideal world - the reality is that people simply keep doing what they do now, and we get the message out via a gradual process of addressing bug reports and providing the explanation there).
I think that this is the most important reason not to go too crazy here. We should think of breaking changes as having a limited budget for introducing breaking changes, and the question then becomes where do we want to spend our budget? While something being relatively new means that the cost to our budget is smaller, it still has a cost and I just don’t think this is a great place to spend our budget at.
On top of that, I don’t think the end result is significantly or meaningfully better for the end users, particularly if we ever get to a world where we isolate everything by default (which would mean that we would have to have an implicit requires of setuptools/wheel anyways).
Fair enough. I just wanted to make the build-system table to be more visible to end users, since I feel the cost here is justified by the benefit s . If others don't agree, that's cool. I don't feel super strongly about this tbh . I am not opposed to keeping the status quo on this matter and am on board with Paul's suggestion of making the build-system table optional. :)
OK, thanks for confirming that. I think everyone who made comments on my proposal was ultimately saying something along the lines of "but I'm not sufficiently bothered to make a fight over it". So in the interests of bringing this discussion to a conclusion and moving on, I'm going to say that we go with what I proposed: 1. It should be legal for pyproject.toml to *not* contain a [build-system] section. 2. If a [build-system] section is present, the requires key must be present. Tools should behave as follows: 1. If [build-system] is present but requires is missing, they should raise an error. 2. If [build-system] is missing, they should take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"] Whether there is any practical difference between cases 2a and 2b is tool-dependent. (It's not ideal that we don't make a recommendation between 2a and 2b, but getting into the details of pip's isolation strategy hasn't really helped make the choice more obvious, and I doubt that any new tools without pip's backward compatibility requirements would behave differently in the two cases anyway). Brett - you offered to update the PEP, so I'll leave that step to you. Thanks to everyone for the helpful discussions and willingness to compromise on the final outcome :-) Paul
Sorry to bump into this so late, but: Would not 2a be more backward compatible than 2b? I mean people may have build environments/installs doing: - pip install setuptools-scm setuptools pbr - pip install os-system-level This way you can bypass the setup requires of setuptools to use easy_install. With 2b there is no way to avoid setup requires elements to not trigger easy install; other than os-system-level (which requires pbr for setup) to switch to pyproject.toml.
OK, thanks for confirming that. I think everyone who made comments on my proposal was ultimately saying something along the lines of "but I'm not sufficiently bothered to make a fight over it". So in the interests of bringing this discussion to a conclusion and moving on, I'm going to say that we go with what I proposed: 1. It should be legal for pyproject.toml to *not* contain a [build-system] section. 2. If a [build-system] section is present, the requires key must be present. Tools should behave as follows: 1. If [build-system] is present but requires is missing, they should raise an error. 2. If [build-system] is missing, they should take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"] Whether there is any practical difference between cases 2a and 2b is tool-dependent. (It's not ideal that we don't make a recommendation between 2a and 2b, but getting into the details of pip's isolation strategy hasn't really helped make the choice more obvious, and I doubt that any new tools without pip's backward compatibility requirements would behave differently in the two cases anyway). Brett - you offered to update the PEP, so I'll leave that step to you. Thanks to everyone for the helpful discussions and willingness to compromise on the final outcome :-) Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/X...
On 19 July 2018 at 11:02, Bernat Gabor <gaborjbernat@gmail.com> wrote:
Sorry to bump into this so late, but:
Would not 2a be more backward compatible than 2b? I mean people may have build environments/installs doing: - pip install setuptools-scm setuptools pbr - pip install os-system-level
This way you can bypass the setup requires of setuptools to use easy_install. With 2b there is no way to avoid setup requires elements to not trigger easy install; other than os-system-level (which requires pbr for setup) to switch to pyproject.toml.
For tools (such as pip) where 2a and 2b behave differently, yes it would. But that's a pip implementation decision which can be made independently of the resolution of this discussion (the PEP allows either behaviour). The details of pip's behaviour have been discussed both on this thread and on the related pip issue that's referenced in this thread. I don't think it's worth rehashing that discussion again here. If you want to argue that pip should switch to behaviour 2a, you should pick up the discussion on the pip issue tracker. But be warned, I'll argue against you over there :-) (Not least because the legacy behaviour is available using --no-build-isolation, which is *precisely* for the case where you want to manage the build tools manually the way you are doing above). Paul
Fair enough :+1: Let's go ahead with this then.
For tools (such as pip) where 2a and 2b behave differently, yes it would. But that's a pip implementation decision which can be made independently of the resolution of this discussion (the PEP allows either behaviour). The details of pip's behaviour have been discussed both on this thread and on the related pip issue that's referenced in this thread. I don't think it's worth rehashing that discussion again here. If you want to argue that pip should switch to behaviour 2a, you should pick up the discussion on the pip issue tracker. But be warned, I'll argue against you over there :-) (Not least because the legacy behaviour is available using --no-build-isolation, which is *precisely* for the case where you want to manage the build tools manually the way you are doing above).
I have updated PEP 518: https://github.com/python/peps/commit/af73627e587c25b9ac6f28a0fda01953252df3... . On Thu, 19 Jul 2018 at 03:29 Bernat Gabor <gaborjbernat@gmail.com> wrote:
Fair enough :+1: Let's go ahead with this then.
For tools (such as pip) where 2a and 2b behave differently, yes it would. But that's a pip implementation decision which can be made independently of the resolution of this discussion (the PEP allows either behaviour). The details of pip's behaviour have been discussed both on this thread and on the related pip issue that's referenced in this thread. I don't think it's worth rehashing that discussion again here. If you want to argue that pip should switch to behaviour 2a, you should pick up the discussion on the pip issue tracker. But be warned, I'll argue against you over there :-) (Not least because the legacy behaviour is available using --no-build-isolation, which is *precisely* for the case where you want to manage the build tools manually the way you are doing above). -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/4...
On Fri, Jul 20, 2018 at 5:01 PM, Brett Cannon <brett@python.org> wrote:
I have updated PEP 518: https://github.com/python/peps/commit/af73627e587c25b9ac6f28a0fda01953252df3...
LGTM. Thanks Brett! -n -- Nathaniel J. Smith -- https://vorpus.org
On 21 July 2018 at 01:15, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Jul 20, 2018 at 5:01 PM, Brett Cannon <brett@python.org> wrote:
I have updated PEP 518: https://github.com/python/peps/commit/af73627e587c25b9ac6f28a0fda01953252df3...
LGTM. Thanks Brett!
LGTM also. Thanks! Paul
On Jul 16, 2018, at 5:22 AM, Paul Moore <p.f.moore@gmail.com> wrote:
1. If [build-system] is present but requires is missing, raise an error. 2. If [build-system] is missing, they can take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"]
Whether tools act differently in cases 2a and 2b is tool-dependent (for pip, we would isolate in case 2b but not in case 2a) which is why the choice is left to individual tools. That makes the "Thomas/Nathaniel" debate into a tool implementation choice, and both of the options are allowable from the perspective of the PEP.
This sounds fine to me, and I prefer a 2b approach.
On Mon, Jul 16, 2018 at 11:27 AM, Donald Stufft <donald@stufft.io> wrote:
On Jul 16, 2018, at 5:22 AM, Paul Moore <p.f.moore@gmail.com> wrote:
1. If [build-system] is present but requires is missing, raise an error. 2. If [build-system] is missing, they can take one of the following two approaches: a) Act as if pyproject.toml is missing altogether b) Act as if [build-system] is present, with a requires value of ["setuptools", "wheel"]
Whether tools act differently in cases 2a and 2b is tool-dependent (for pip, we would isolate in case 2b but not in case 2a) which is why the choice is left to individual tools. That makes the "Thomas/Nathaniel" debate into a tool implementation choice, and both of the options are allowable from the perspective of the PEP.
This sounds fine to me, and I prefer a 2b approach.
I also prefer option 2 (and specifically 2b but like you say, 2a vs 2b isn't something the PEP cares about), just because it's the simplest possible approach: we always act the same when build-system.requires is missing, regardless of why it's missing. And it's the same logic as we use to handle a missing build-system.build-backend. It doesn't matter that much though. It seems extremely unlikely that anyone's going to create an empty [build-system] section just because they can... -n -- Nathaniel J. Smith -- https://vorpus.org
On 22 June 2018 at 18:01, Brett Cannon <brett@python.org> wrote:
On Fri, 22 Jun 2018 at 09:35 Pradyun Gedam <pradyunsg@gmail.com> wrote:
[...]
I think the precise wording is there by not having any wording. ;) If 'requires' was meant to be optional then the PEP would have said that.
FWIW, that's my view too.
I can think of at least 2 options for behavior when build-system.requires is missing:
1. Consider a missing build-system.requires equivalent to either a missing pyproject.toml or build-system.requires = ["setuptools", "wheel"].
2. Making the build-system table mandatory in pyproject.toml.
I personally think (2) would be fine -- "Explicit is better than implicit."
And I think that's what pip's warning is saying the future will be, but they aren't quite yet ready to be that strict yet with their users (which I can understand).
Precisely. We currently do accept a missing key, but the presence of pyproject.toml triggers a user-visible change in behaviour (build isolation). That is how this question came to light. The current warning is precisely intended as a deprecation warning that we're heading towards implementing (2) but we want to give people a chance to prepare. The main concern is that tools like towncrier and black *only* allow configuration to be specified in pyproject.toml, meaning that projects using those tools are using that file just as a tool config file, rather than for its stated purpose of build system configuration. IMO, that's an unfortunate choice that's been made by those tools (essentially, they have interpreted pyproject.toml as a "common location for configuration data" rather than a build config file) but that's their choice. Projects can pretty easily address the warning just by adding [build-system] requires = ["setuptools", "wheel"] to their pyproject.toml (that's what the pip project itself has done). The only case where this is an issue is where the project simultaneously needs to avoid the new build isolation mechanism - and honestly I think that's both a pretty rare situation, and not something we want to be supporting long-term anyway. I've started referring to the "no pyproject.toml, no build isolation" behaviour as the "legacy code path" to emphasise that it's only retained to support projects that haven't yet moved to the newer standards. Paul
On Fri, 22 Jun 2018 at 12:51 Paul Moore <p.f.moore@gmail.com> wrote:
On 22 June 2018 at 18:01, Brett Cannon <brett@python.org> wrote:
On Fri, 22 Jun 2018 at 09:35 Pradyun Gedam <pradyunsg@gmail.com> wrote:
[...]
I think the precise wording is there by not having any wording. ;) If 'requires' was meant to be optional then the PEP would have said that.
FWIW, that's my view too.
I can think of at least 2 options for behavior when build-system.requires is missing:
1. Consider a missing build-system.requires equivalent to either a missing pyproject.toml or build-system.requires = ["setuptools", "wheel"].
2. Making the build-system table mandatory in pyproject.toml.
I personally think (2) would be fine -- "Explicit is better than implicit."
And I think that's what pip's warning is saying the future will be, but they aren't quite yet ready to be that strict yet with their users (which I can understand).
Precisely. We currently do accept a missing key, but the presence of pyproject.toml triggers a user-visible change in behaviour (build isolation). That is how this question came to light. The current warning is precisely intended as a deprecation warning that we're heading towards implementing (2) but we want to give people a chance to prepare.
The main concern is that tools like towncrier and black *only* allow configuration to be specified in pyproject.toml, meaning that projects using those tools are using that file just as a tool config file, rather than for its stated purpose of build system configuration. IMO, that's an unfortunate choice that's been made by those tools (essentially, they have interpreted pyproject.toml as a "common location for configuration data" rather than a build config file) but that's their choice.
I view it as Black and Towncrier consider themselves tools as part of the build process and now helped us push pyproject.toml faster. :)
Projects can pretty easily address the warning just by adding
[build-system] requires = ["setuptools", "wheel"]
to their pyproject.toml (that's what the pip project itself has done). The only case where this is an issue is where the project simultaneously needs to avoid the new build isolation mechanism - and honestly I think that's both a pretty rare situation, and not something we want to be supporting long-term anyway. I've started referring to the "no pyproject.toml, no build isolation" behaviour as the "legacy code path" to emphasise that it's only retained to support projects that haven't yet moved to the newer standards.
Yeah, fixing the problem is two lines of boilerplate, so hopefully the complains should be minimal.
To go a bit against the grain here, I think at this point I'd suggest that if "build-system.requires" is missing, it should be silently treated as if it had been set to ["setuptools", "wheel"]. Reasoning: - Implementing this should require only a trivial amount of code, now and in the long run. In particular, I'm *not* suggesting that if the "build-system.requires" key is missing then we should act like pyproject.toml is missing altogether -- that's a much more complex legacy code path that we'd like to eventually remove. I'm suggesting we literally do something like: try: requires = config["build-system"]["requires"] except KeyError: requires = ["setuptools", "wheel"] and then treat them exactly the same from then on. - Not doing this breaks a number of real projects. Sometimes this is justifiable because we have to break things to make progress, but it always creates busywork and pisses people off, so we should only do it when we have a good reason. In this case providing a default value is pretty trivial, and will prevent a lot of frustrated queries about why it's mandatory. - Providing a default doesn't really compromise the final vision for the feature: we envision that eventually, pretty much every project will be specifying this explicitly, and won't *want* to leave it blank. There isn't any other meaning we want to assign to this being left blank. - We're soon going to have to jump through all these hoops *anyway* for the PEP 517 "build-system.build-backend" key. If it's missing, then we're going to want to default it to "setuptools" (once setuptools exports a PEP 517 build backend), which means we're going to be hardcoding some defaults and knowledge of setuptools into the pyproject.toml defaults. So we might as well do this for both keys in the same way. -n On Fri, Jun 22, 2018 at 9:32 AM, Pradyun Gedam <pradyunsg@gmail.com> wrote:
Hey everyone!
In PEP 518, it is not clearly specified how a project that has a pyproject.toml file but has no build-system.requires should be treated (i.e. build-system table).
In pip 10, such a pyproject.toml file was allowed and built with setuptools and wheel, which has resulted in a lot of projects making releases that assumed that such a pyproject.toml file is valid and they use setuptools and wheel. I understand that at least pytest, towncrier and Twisted might have done so. This happened since these projects have included configuration for some tool in pyproject.toml (some of which use only pyproject.toml for configuration -- black, towncrier).
There's a little bit of subtlety here, in pip 10's implementation: adding a pyproject.toml file enables a new code path that does the build in isolation (in preparation for PEP 517; it's a good idea on it's own too) with only the build-system.requires packages available. When the build-system.requires key is missing, pip falls back to assuming it should be ["setuptools", "wheel"]. The in-development version of pip currently prints warnings when the key is not specified -- along the lines of "build-system.requires is missing" + "A future version of pip will reject pyproject.toml files that do not comply with PEP 518." and falls back to legacy behavior.
Basically, pip 10 has a distinction between a missing pyproject.toml and build-system.requires = ["setuptools", "wheel"] and the PEP doesn't. However, the PEP's precise wording here would help inform the debate about how pip should behave in this edge case.
I can think of at least 2 options for behavior when build-system.requires is missing:
1. Consider a missing build-system.requires equivalent to either a missing pyproject.toml or build-system.requires = ["setuptools", "wheel"].
2. Making the build-system table mandatory in pyproject.toml.
I personally think (2) would be fine -- "Explicit is better than implicit."
It'll be easy to detect and error out in this case, in a way that it's possible to provide meaningful information to the user about what to do here. However, this does mean that some existing releases of projects become not-installable, which is concerning; I do think the benefits outweigh the costs though.
Thoughts on this?
Cheers, Pradyun
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/7...
-- Nathaniel J. Smith -- https://vorpus.org
On Sun, Jun 24, 2018 at 10:50 AM Nathaniel Smith <njs@pobox.com> wrote:
To go a bit against the grain here, I think at this point I'd suggest that if "build-system.requires" is missing, it should be silently treated as if it had been set to ["setuptools", "wheel"]. Reasoning:
- Implementing this should require only a trivial amount of code, now and in the long run. In particular, I'm *not* suggesting that if the "build-system.requires" key is missing then we should act like pyproject.toml is missing altogether -- that's a much more complex legacy code path that we'd like to eventually remove. I'm suggesting we literally do something like:
try: requires = config["build-system"]["requires"] except KeyError: requires = ["setuptools", "wheel"]
and then treat them exactly the same from then on.
Defaulting to this behavior means that the way source distribution is built changes (build isolation is enabled by pip) because configuration for a tool was added. This is surprising for users since one of things this means is they need to have provide wheels (so that pip that can find/use them) for `setuptools` and `wheel` to install packages. We've had multiple users report this on pip's tracker. Having users specify their build-requirements explicitly is a stronger opt-in that can used to explain the behavior in this case as that project using PEP 518 and build-isolation vs that project has configuration for towncrier. :) The only other option is falling back to legacy behavior in this case, which obviously isn't what we want here. - Not doing this breaks a number of real projects. Sometimes this is
justifiable because we have to break things to make progress, but it always creates busywork and pisses people off, so we should only do it when we have a good reason. In this case providing a default value is pretty trivial, and will prevent a lot of frustrated queries about why it's mandatory.
- Providing a default doesn't really compromise the final vision for the feature: we envision that eventually, pretty much every project will be specifying this explicitly, and won't *want* to leave it blank. There isn't any other meaning we want to assign to this being left blank.
- We're soon going to have to jump through all these hoops *anyway* for the PEP 517 "build-system.build-backend" key. If it's missing, then we're going to want to default it to "setuptools" (once setuptools exports a PEP 517 build backend), which means we're going to be hardcoding some defaults and knowledge of setuptools into the pyproject.toml defaults. So we might as well do this for both keys in the same way.
-n
Pradyun
On Wed, Jun 27, 2018 at 8:00 AM, Pradyun Gedam <pradyunsg@gmail.com> wrote:
On Sun, Jun 24, 2018 at 10:50 AM Nathaniel Smith <njs@pobox.com> wrote:
To go a bit against the grain here, I think at this point I'd suggest that if "build-system.requires" is missing, it should be silently treated as if it had been set to ["setuptools", "wheel"]. Reasoning:
- Implementing this should require only a trivial amount of code, now and in the long run. In particular, I'm *not* suggesting that if the "build-system.requires" key is missing then we should act like pyproject.toml is missing altogether -- that's a much more complex legacy code path that we'd like to eventually remove. I'm suggesting we literally do something like:
try: requires = config["build-system"]["requires"] except KeyError: requires = ["setuptools", "wheel"]
and then treat them exactly the same from then on.
Defaulting to this behavior means that the way source distribution is built changes (build isolation is enabled by pip) because configuration for a tool was added. This is surprising for users since one of things this means is they need to have provide wheels (so that pip that can find/use them) for `setuptools` and `wheel` to install packages. We've had multiple users report this on pip's tracker.
I'm not sure I understand the specific surprise you're talking about -- are you saying it's common for people to somehow find themselves in environments where setuptools and wheel are not installable?
Having users specify their build-requirements explicitly is a stronger opt-in that can used to explain the behavior in this case as that project using PEP 518 and build-isolation vs that project has configuration for towncrier. :)
The only other option is falling back to legacy behavior in this case, which obviously isn't what we want here.
This does make some sense. I'm pondering it :-). To make sure I understand, you're thinking of a scenario like: <End user> Your software is broken! Installing it used to work fine, but now it's giving me an error saying 'can't install setuptools'! <Dev> Uh, that's weird, I'm not sure why that happens. Let me look into it. [...time passes...] <Dev> Okay, I figured it out: it's because we added a pyproject.toml file, and that switches on this 'build isolation' thing, which shouldn't make a difference, unless... does 'pip install setuptools' work for you? <End user> Oh yeah I didn't think it was worth mentioning, but we have a broken firewall where we have to manually download each package and install it one at a time. Do you think my not being able to install setuptools might have something to do with getting an error saying 'can't install setuptools'? <Dev> It's possible, yeah. <End user> So how are you going to fix it? <Dev> Well, we're not going to stop using towncrier because your firewall is broken, and even if we did then you'd have the same problem with like, every other Python package you ever want to install, so you should probably fix stuff on your end. Good luck. And in particular, you're talking about trying to optimize that '[...time passes...]' part at the beginning: if the dev added a pyproject.toml just to use towncrier and never even realized that [build-system] sections are a thing that exists, then it might take them a while to figure out that the error message and the pyproject.toml file could be related. OTOH, if they were forced to type '[build-system] requires = ["setuptools", "wheel"]', then it increases the chances that a few months later when they get this request from a user, they'll think "hey, wasn't there something about setuptools that I touched recently.....?" and that will speed up their figuring it out. Have I understood all that correctly? That all sounds like something that could happen, but it still feels a bit... tenuous, or something? If I were the dev here then there's an excellent chance that I'd have totally forgotten about that '[build-system]' stuff before the bug report comes in, and even if I haven't, the connection to their problem isn't obvious, and in any case knowing the issue doesn't really change the conclusion -- if someone can't install setuptools then that's the thing they need to fix. If this is a common scenario, then isn't the real solution for pip to print a better error message in the first place, so that the dev doesn't need to debug this from scratch? -n -- Nathaniel J. Smith -- https://vorpus.org
participants (10)
-
Bernat Gabor -
Brett Cannon -
Daniel Shaulov -
Donald Stufft -
Jeremy Stanley -
Nathaniel Smith -
Nick Coghlan -
Paul Moore -
Pradyun Gedam -
Thomas Kluyver