Does ensurepip still have to include a copy of setuptools?
Hi everyone, ensurepip includes private copies of pip and setuptools. But PEP 453 states that "once pip is able to run pip install --upgrade pip without needing setuptools installed first, then the private copy of setuptools will be removed from ensurepip in subsequent CPython releases." https://www.python.org/dev/peps/pep-0453/#automatic-installation-of-setuptoo... At the moment pip itself includes a needed part of setuptools. https://github.com/pypa/pip/tree/9c474d4862907ae220ced0fcdbd76660955ff732/sr... I experimented with modifying ensurepip in the main branch not to install setuptools, and then used it to install pip. It worked fine. Then I run `./python -m pip install --upgrade pip`, and it upgraded pip successfully. Does this mean that we can drop the copy of setuptools? Note, the venv module has such code `CORE_VENV_DEPS = ('pip', 'setuptools')`. I am not sure whether it requires setuptools from ensurepip. Thanks, Illia
On Sat, 2 Oct 2021 at 03:27, Illia Volochii <illia.volochii@gmail.com> wrote:
Interesting. Pip does not need setuptools installed to upgrade itself, so a strict reading of the PEP would seem to imply that we need to remove setuptools, as you say. However, looking at things more practically, pip still needs setuptools to do "legacy" installs of source distributions (when the project does not include a `pyproject.toml`) and for editable installs of setuptools-based projects. If we stopped shipping setuptools as part of ensurepip, I imagine people would complain that we'd "broken" things. Telling them that it's not broken, all they need to do is `pip install setuptools`, and we only ever promised that the supplied pip could be used to bootstrap a full environment, doesn't seem likely to go down well IMO. (Technically, some aspects of pip don't work, or fall back to "legacy" code paths, if the `wheel` project isn't installed, so ensurepip "needs" wheel in the same sense as it "needs" setuptools, but the breakage is less significant, and people who care are used to the current situation and know what to do. So yes, that argues we could do the same to setuptools, it's just a bigger impact.)
At the moment pip itself includes a needed part of setuptools. https://github.com/pypa/pip/tree/9c474d4862907ae220ced0fcdbd76660955ff732/sr...
That's internal to pip, and the pip code that uses that, does not need an externally-supplied setuptools.
IMO, it's too early to consider dropping setuptools, notwithstanding what a strict reading of the PEP says. When pip has removed more of the "legacy" code paths, this situation could change, but we're not there yet. Paul
On Sat, 2 Oct 2021 at 12:20, Thomas Grainger <tagrain@gmail.com> wrote:
I raised an issue about this: https://github.com/pypa/pip/issues/10530
I agree with the comment made on that issue - this isn't the right way to handle the problem. We need to encourage projects to opt into the new approach and remove the legacy path once it's no longer needed. We should *not* maintain the "old style" approach indefinitely, hiding the fact that it's no longer the correct approach by having some sort of "auto convert" logic in the tools. Doing that has the *opposite* effect to what we're trying to achieve - adoption of cleaner modern approaches will take *longer*, because we're actively allowing projects to continue using their existing approach with no consequences. Paul
On Sat, 2 Oct 2021 13:27:03 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
Can you explain what the "old style" approach is here? I would hope for the "old style" approach to be deprecated (with a *visible* warning message) for at least 2 years before it is removed. It is nice that well-maintained packages with lots of contributors get frequent releases and keep up with the pace of changes in the packaging ecosystem, but please don't forget that there's a long tail of packages that are updated infrequently and but still work properly and perform an important function for some parts of the user base. Regards Antoine.
On Mon, 4 Oct 2021 at 08:21, Antoine Pitrou <antoine@python.org> wrote:
I'm talking about projects adopting a `pyproject.toml` configuration file that specifies the build backend to use (in this case setuptools). The `pyproject.toml` style was standardised in PEP 517 about 4 years ago now, and projects have been gradually adopting it. As you say, there's a long tail of projects who have no immediate need to switch, but we're working on smoothing that transition as much as we can. Deprecation is a complex process, and not really a python-dev question, but for the record, pip (and PEP 517) have a mechanism for using the new-style hooks even for older projects that haven't adopted it. That, plus a "build isolation" mechanism, allows pip to work even if setuptools is not present. We're transitioning to making that the default behaviour, but that process isn't yet complete. Although even when it *is* complete, we may have options allowing use of the old behaviour (`--no-build-isolation, --no-use-pep517) for some time after that. Regarding warning when the old `setup.py` mechanism is used instead of the new PEP 517 hooks, that's a matter for setuptools to decide, and I can't speak for them. It's also not relevant to when ensurepip drops inclusion of setuptools, as the ensurepip requirement is only that *pip* no longer needs setuptools, and as I said, we're hoping to make that mostly transparent. We *might* also be able to add a warning in pip, to catch the case where setuptools *doesn't* have the warning, but honestly we will probably be mostly OK from our side of things with just advising the user to install setuptools manually in the (increasingly rare) cases where we can't work out to install it automatically.
We (the packaging community) are *extremely* aware of this, yes. If you're interested in helping out, then these sorts of discussions happen on the packaging area in Discourse, and (for project-specific items) on the pip and setuptools trackers. We'd love more help there - packaging for Python is critically under-resourced! - so please feel welcome to come along and join in the work :-) Paul
On Mon, 4 Oct 2021, 5:23 pm Antoine Pitrou, <antoine@python.org> wrote:
That's the real milestone intended in the setuptools bundling wording: we can drop the bundling when pip doesn't need it itself, *and* can also transparently bootstrap it for all other projects that need it, whether those projects have been updated to use the modern packaging standards or not. The fact those were different requirements wasn't clear when we wrote the PEP though, so the concrete example given only covers the first criterion and not the second one, rendering the section as a whole ambiguous. Paul's mail covered some of the details still pending before automatic installation of build time dependencies into an isolated build environment becomes a universal default on new pip installations. We wouldn't drop setuptools from the bundling until that default changes. Cheers, Nick.
On Sat, 2 Oct 2021 at 03:27, Illia Volochii <illia.volochii@gmail.com> wrote:
Interesting. Pip does not need setuptools installed to upgrade itself, so a strict reading of the PEP would seem to imply that we need to remove setuptools, as you say. However, looking at things more practically, pip still needs setuptools to do "legacy" installs of source distributions (when the project does not include a `pyproject.toml`) and for editable installs of setuptools-based projects. If we stopped shipping setuptools as part of ensurepip, I imagine people would complain that we'd "broken" things. Telling them that it's not broken, all they need to do is `pip install setuptools`, and we only ever promised that the supplied pip could be used to bootstrap a full environment, doesn't seem likely to go down well IMO. (Technically, some aspects of pip don't work, or fall back to "legacy" code paths, if the `wheel` project isn't installed, so ensurepip "needs" wheel in the same sense as it "needs" setuptools, but the breakage is less significant, and people who care are used to the current situation and know what to do. So yes, that argues we could do the same to setuptools, it's just a bigger impact.)
At the moment pip itself includes a needed part of setuptools. https://github.com/pypa/pip/tree/9c474d4862907ae220ced0fcdbd76660955ff732/sr...
That's internal to pip, and the pip code that uses that, does not need an externally-supplied setuptools.
IMO, it's too early to consider dropping setuptools, notwithstanding what a strict reading of the PEP says. When pip has removed more of the "legacy" code paths, this situation could change, but we're not there yet. Paul
On Sat, 2 Oct 2021 at 12:20, Thomas Grainger <tagrain@gmail.com> wrote:
I raised an issue about this: https://github.com/pypa/pip/issues/10530
I agree with the comment made on that issue - this isn't the right way to handle the problem. We need to encourage projects to opt into the new approach and remove the legacy path once it's no longer needed. We should *not* maintain the "old style" approach indefinitely, hiding the fact that it's no longer the correct approach by having some sort of "auto convert" logic in the tools. Doing that has the *opposite* effect to what we're trying to achieve - adoption of cleaner modern approaches will take *longer*, because we're actively allowing projects to continue using their existing approach with no consequences. Paul
On Sat, 2 Oct 2021 13:27:03 +0100 Paul Moore <p.f.moore@gmail.com> wrote:
Can you explain what the "old style" approach is here? I would hope for the "old style" approach to be deprecated (with a *visible* warning message) for at least 2 years before it is removed. It is nice that well-maintained packages with lots of contributors get frequent releases and keep up with the pace of changes in the packaging ecosystem, but please don't forget that there's a long tail of packages that are updated infrequently and but still work properly and perform an important function for some parts of the user base. Regards Antoine.
On Mon, 4 Oct 2021 at 08:21, Antoine Pitrou <antoine@python.org> wrote:
I'm talking about projects adopting a `pyproject.toml` configuration file that specifies the build backend to use (in this case setuptools). The `pyproject.toml` style was standardised in PEP 517 about 4 years ago now, and projects have been gradually adopting it. As you say, there's a long tail of projects who have no immediate need to switch, but we're working on smoothing that transition as much as we can. Deprecation is a complex process, and not really a python-dev question, but for the record, pip (and PEP 517) have a mechanism for using the new-style hooks even for older projects that haven't adopted it. That, plus a "build isolation" mechanism, allows pip to work even if setuptools is not present. We're transitioning to making that the default behaviour, but that process isn't yet complete. Although even when it *is* complete, we may have options allowing use of the old behaviour (`--no-build-isolation, --no-use-pep517) for some time after that. Regarding warning when the old `setup.py` mechanism is used instead of the new PEP 517 hooks, that's a matter for setuptools to decide, and I can't speak for them. It's also not relevant to when ensurepip drops inclusion of setuptools, as the ensurepip requirement is only that *pip* no longer needs setuptools, and as I said, we're hoping to make that mostly transparent. We *might* also be able to add a warning in pip, to catch the case where setuptools *doesn't* have the warning, but honestly we will probably be mostly OK from our side of things with just advising the user to install setuptools manually in the (increasingly rare) cases where we can't work out to install it automatically.
We (the packaging community) are *extremely* aware of this, yes. If you're interested in helping out, then these sorts of discussions happen on the packaging area in Discourse, and (for project-specific items) on the pip and setuptools trackers. We'd love more help there - packaging for Python is critically under-resourced! - so please feel welcome to come along and join in the work :-) Paul
On Mon, 4 Oct 2021, 5:23 pm Antoine Pitrou, <antoine@python.org> wrote:
That's the real milestone intended in the setuptools bundling wording: we can drop the bundling when pip doesn't need it itself, *and* can also transparently bootstrap it for all other projects that need it, whether those projects have been updated to use the modern packaging standards or not. The fact those were different requirements wasn't clear when we wrote the PEP though, so the concrete example given only covers the first criterion and not the second one, rendering the section as a whole ambiguous. Paul's mail covered some of the details still pending before automatic installation of build time dependencies into an isolated build environment becomes a universal default on new pip installations. We wouldn't drop setuptools from the bundling until that default changes. Cheers, Nick.
participants (5)
-
Antoine Pitrou -
Illia Volochii -
Nick Coghlan -
Paul Moore -
Thomas Grainger