PEP 632: Deprecate distutils module
Hi all. setuptools has recently adopted the entire codebase of the distutils module, so that they will be able to make improvements directly without having to rely on patching the standard library. As a result, we can now move forward with official deprecation (in 3.10) and removal (in 3.12) (noting that the distutils docs already recommend switching to setuptools). Full text and discussion at https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134 I'm including the original text below, but won't be responding to all discussions here (though I'll periodically check in and skim read it, assuming things don't go way off track). Also be aware that I already have some minor changes lined up that are not in this text. Refer to the discussion on Discourse if you need to see those. Cheers, Steve --- PEP: 632 Title: Deprecate distutils module Author: Steve Dower <steve.dower@python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 03-Sep-2020 Post-History: Abstract ======== The distutils module [1]_ has for a long time recommended using the setuptools package [2]_ instead. Setuptools has recently integrated a complete copy of distutils and is no longer dependent on the standard library [3]_. Pip has silently replaced distutils with setuptools when building packages for a long time already. It is time to remove it from the (public part of the) standard library. Motivation ========== distutils [1]_ is a largely undocumented and unmaintained collection of utilities for packaging and distributing Python packages, including compilation of native extension modules. It defines a configuration format that describes a Python distribution and provides the tools to convert a directory of source code into a source distribution, and some forms of binary distribution. Because of its place in the standard library, many updates can only be released with a major release, and users cannot rely on particular fixes being available. setuptools [2]_ is a better documented and well maintained enhancement based on distutils. While it provides very similar functionality, it is much better able to support users on earlier Python releases, and can respond to bug reports more quickly. A number of platform-specific enhancements already exist in setuptools that have not been added to distutils, and there is been a long-standing recommendation in the distutils documentation to prefer setuptools. Historically, setuptools has extended distutils using subclassing and monkeypatching, but has now taken a copy of the underlying code. [3]_ As a result, the second last major dependency on distutils is gone and there is no need to keep it in the standard library. The final dependency on distutils is CPython itself, which uses it to build the native extension modules in the standard library (except on Windows). Because this is a CPython build-time dependency, it is possible to continue to use distutils for this specific case without it being part of the standard library. Deprecation and removal will make it obvious that issues should be fixed in the setuptools project, and will reduce a source of bug reports and test maintenance that is unnecessary. It will also help promote the development of alternative build backends, which can now be supported more easily thanks to PEP 517. Specification ============= In Python 3.10 and 3.11, distutils will be formally marked as deprecated. All known issues will be closed at this time. ``import distutils`` will raise a deprecation warning. During Python 3.10 and 3.11, uses of distutils within the standard library may change to use alternative APIs. In Python 3.12, distutils will no longer be installed by ``make install`` or any of the first-party distribution. Third-party redistributors should no longer include distutils in their bundles or repositories. This PEP makes no specification on migrating the parts of the CPython build process that currently use distutils. Depending on contributions, this migration may occur at any time. After Python 3.12 is started and when the CPython build process no longer depends on distutils being in the standard library, the entire ``Lib/distutils`` directory and ``Lib/test/test_distutils.py`` file will be removed from the repository. Other references to distutils will be cleaned up. As of Python 3.9's initial release, the following modules have references in code or comments: * Lib/ctypes/util.py * Lib/site.py * Lib/sysconfig.py * Lib/_aix_support.py * Lib/_bootsubprocess.py * Lib/_osx_support.py * Modules/_decimal/tests/formathelper.py As the distutils code is already included in setuptools, there is no need to republish it in any other form. Those who require access to the functionality should use setuptools or an alternative build backend. Backwards Compatibility ======================= Code that imports distutils will no longer work from Python 3.12. The suggested migration path is to use the equivalent (though not identical) imports from setuptools. Compatibility shims already exist in setuptools and pip to transparently migrate old code, however, these are controlled by external projects and are in no way bound by this PEP. Consult their latest documentation for migration advice. Some projects use alternate sets of shims over distutils, notably, numpy.distutils. [4]_ Where known, these projects have been informed. Many use custom commands or more narrowly scoped patches, mostly constrained to ``setup.py`` files. As these packages are already subject to setuptools being injected by pip, we expect minimal disruption as a result of distutils being removed. Reference Implementation ======================== setuptools version 48 includes the complete copy of distutils, and as such is no longer dependent on the standard library's copy. Most implementation issues they have faced are due to the continuing existence of distutils in the standard library, and so removal will improve the stability of their implementation. There is not yet a reference implementation for the removal of distutils from the standard library, nor is there an implementation for CPython's native module builds without relying on the standard library copy of distutils. References ========== .. [1] distutils - Building and installing Python modules (https://docs.python.org/3.9/library/distutils.html) .. [2] setuptools - PyPI (https://pypi.org/project/setuptools/) .. [3] setuptools Issue #417 - Adopt distutils (https://github.com/pypa/setuptools/issues/417) .. [4] Packaging (numpy.distutils) (https://numpy.org/doc/stable/reference/distutils.html) Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
Hi, Aha, someone proposed the idea :-) I know that Debian removed distutils from Python stdlib (it is a separate package) for a long time. You may mention https://github.com/pypa/distutils/ somewhere in the PEP: "Python Module Distribution Utilities extracted from the Python Standard Library". I don't know the future of this project since setuptools now also contains distutils. I would be nice to hear about setuptools maintainers here. setuptools 50.0.0 release didn't go well: it broke many use cases on Debian, Ubuntu and Fedora, etc. See: https://github.com/pypa/setuptools/issues especially https://github.com/pypa/setuptools/issues/2350 For example, Fedora does patch the stdlib distutils with a downstream-only patch, so pip installs into /usr/local rather than /usr. Well, there is likely a way to fix this issue in setuptools, but right now the situation is complicated. Le ven. 4 sept. 2020 à 13:37, Steve Dower <steve.dower@python.org> a écrit :
The final dependency on distutils is CPython itself, which uses it to build the native extension modules in the standard library (except on Windows). Because this is a CPython build-time dependency, it is possible to continue to use distutils for this specific case without it being part of the standard library. (...) After Python 3.12 is started and when the CPython build process no longer depends on distutils being in the standard library, the entire Lib/distutils directory and Lib/test/test_distutils.py file will be removed from the repository.
In practical terms, how Python 3.12 will build its extensions if it doesn't contain distutils anymore? Should we vendor a copy of setuptools, just for Python setup.py? So far, Python has no external dependency to be built. It makes it easy to build on various platforms. I would prefer to not have to download something from the Internet to build Python after I downloaded a Python tarball. We already have wheel packages of setuptools and pip in Lib/ensurepip/_bundled/. Victor
On Fri, 4 Sep 2020 12:28:30 +0100 Steve Dower <steve.dower@python.org> wrote:
It will also help promote the development of alternative build backends, which can now be supported more easily thanks to PEP 517.
While I agree with the general suggestion of deprecating distutils as a publicly-visible module (in favour of encouraging users to rely on setuptools), it seems to me that the argument of facilitating the development of third-party build systems is what already encouraged the official policy of not adding features to distutils (more than 10 years ago, IIRC). AFAIK, we're still waiting for those other build systems to appear. But, who knows, this time it will be different. Regards Antoine.
On Fri, Sep 4, 2020 at 11:02 AM Antoine Pitrou <solipsis@pitrou.net> wrote:
While I agree with the general suggestion of deprecating distutils as a publicly-visible module (in favour of encouraging users to rely on setuptools), it seems to me that the argument of facilitating the development of third-party build systems is what already encouraged the official policy of not adding features to distutils (more than 10 years ago, IIRC).
My recollection is that we decided to stop changing distutils because too many 3rd party extensions (often included directly in the packages that needed them) using undocumented parts fo distutils directly (since there was no substantially documented API for distutils in the beginning). Every time anything changed, something was broken for somebody. Since that affected not only the developers hooking in to distutils but the users of their packages, touching distutils caused too much pain. -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein
On Fri, 4 Sep 2020 at 16:53, Fred Drake <fred@fdrake.net> wrote:
On Fri, Sep 4, 2020 at 11:02 AM Antoine Pitrou <solipsis@pitrou.net> wrote:
While I agree with the general suggestion of deprecating distutils as a publicly-visible module (in favour of encouraging users to rely on setuptools), it seems to me that the argument of facilitating the development of third-party build systems is what already encouraged the official policy of not adding features to distutils (more than 10 years ago, IIRC).
My recollection is that we decided to stop changing distutils because too many 3rd party extensions (often included directly in the packages that needed them) using undocumented parts fo distutils directly (since there was no substantially documented API for distutils in the beginning). Every time anything changed, something was broken for somebody. Since that affected not only the developers hooking in to distutils but the users of their packages, touching distutils caused too much pain.
I believe that's correct. My main concern here is that removing distutils from the stdlib won't have made that problem any better, it will simply have transferred the responsibility onto the setuptools maintainers. While I assume that they are comfortable with that, I suspect they may take a different position on backwards compatibility than core Python does (not least because one copy of setuptools has to support multiple python versions, including alternative versions like PyPy, whereas the stdlib copy only needs to handle the version of Python it's distributed with). I think the arguments in favour of this PEP from CPython's point of view are fairly strong, but the arguments from the point of view of the wider Python ecosystem are much less clear. Paul
I believe that's correct. My main concern here is that removing distutils from the stdlib won't have made that problem any better, it will simply have transferred the responsibility onto the setuptools maintainers. While I assume that they are comfortable with that, I suspect they may take a different position on backwards compatibility than core Python does (not least because one copy of setuptools has to support multiple python versions, including alternative versions like PyPy, whereas the stdlib copy only needs to handle the version of Python it's distributed with). I think that it's /basically/ true that this move does transfer that responsibility over to setuptools, and I'm pretty sure that this is effectively handing over a big deprecation to deprecation specialists, since — at least as long as I've been involved with it — the process of
On 9/4/20 12:22 PM, Paul Moore wrote: maintaining setuptools is dominated by deprecating things (we do bugfixes and add features as well, of course, but there's a lot more to deprecate than in your typical project). That said, there are two major advantages to moving distutils into setuptools as the first step in making these "backwards-incompatible" changes (moving distutils into PyPI has similar advantages): 1. Deprecation notices start going out to /all/ supported versions of Python /immediately/ if they are using setuptools, making it easier to get the ecosystem to move together. 2. The fact that setuptools supports many versions of Python decouples the upgrade cycle of setuptools from the upgrade cycle of Python. Users can opt-in to new features by pinning a minimum version of setuptools (allowing them to take on migrations /without/ needing to upgrade their Python version), and if setuptools removes a feature they are counting on, they can pin a maximum version of setuptools mostly without disrupting their ability to upgrade Python versions. Related to this, setuptools can (and does) do more frequent updates, so it's easier to make a quick release undoing major breaking changes (or adding a new feature to work around them, etc).
I think the arguments in favour of this PEP from CPython's point of view are fairly strong, but the arguments from the point of view of the wider Python ecosystem are much less clear.
I mostly agree that this is more useful for the people maintaining setuptools and distutils than it is for consumers of these packages, though that's not necessarily a bad thing. The downside is that we're going to make a bunch of breaking changes over the next few years (hopefully well-documented and with clear migration paths). The upside is that it will be easier for people to reap the benefits of the work we're doing to improve the packaging ecosystem (standardized build artifacts, bugfixes, etc).
Paul _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QRCA2AHO... Code of Conduct: http://python.org/psf/codeofconduct/
I have been using standard distutils for large C extensions without any issues. Since distutils does not change, why remove it? It is a lot of work for people with little gain. I'd really like to build C extensions without downloading an external package. Features like C++ support have not been worked on for more than a decade. Are the setuptools maintainers planning to address these issues now?
* Modules/_decimal/tests/formathelper.py
elif find_executable('locale'): locale_list = subprocess.Popen(["locale", "-a"], stdout=subprocess.PIPE).communicate()[0] One of the many things that just work out of the box. -10 on removing distutils from the stdlib. Freezing it is fine. Stefan Krah
On 9/4/20 12:45 PM, Stefan Krah wrote:
Since distutils does not change, why remove it? It is a lot of work for people with little gain.
I'd really like to build C extensions without downloading an external package. How often do you actually build extensions without building or installing external packages? You don't use `pip install` or PEP 517 builds? Just legacy build and installs? Do you not build or release wheels (which requires the `wheel` package)? Are you planning to upload artifacts to PyPI — if so, won't you need an external package (or at least a maintained package that can keep up with the APIs? Before we deprecated and removed it in setuptools, setup.py upload was causing
If we don't remove it, we should at least freeze the bug component so that people cannot report bugs in distutils. Triaging these bugs alone is a decent amount of work. We should probably also set up a Bedevere to auto-reject PRs that touch distutils files (since telling people that distutils is frozen and no longer maintained is effort as well), and disable distutils in the CI so that it does not generate work for people maintaining the buildbots. problems with the metadata it uploaded — we may need to ban distutils-created packages from PyPI in order to keep PyPI going).
Features like C++ support have not been worked on for more than a decade. Are the setuptools maintainers planning to address these issues now?
Considering that we /aren't/ adding anything to distutils today, the chances of this happening in setuptools are pretty much strictly better than in distutils.
* Modules/_decimal/tests/formathelper.py elif find_executable('locale'): locale_list = subprocess.Popen(["locale", "-a"], stdout=subprocess.PIPE).communicate()[0]
One of the many things that just work out of the box. -10 on removing distutils from the stdlib. Freezing it is fine.
Stefan Krah
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6LQN5OAJ... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Sep 04, 2020 at 01:10:37PM -0400, Paul Ganssle wrote:
On 9/4/20 12:45 PM, Stefan Krah wrote:
Since distutils does not change, why remove it? It is a lot of work for people with little gain.
If we don't remove it, we should at least freeze the bug component so that people cannot report bugs in distutils. Triaging these bugs alone is a decent amount of work. We should probably also set up a Bedevere to auto-reject PRs that touch distutils files (since telling people that distutils is frozen and no longer maintained is effort as well), and disable distutils in the CI so that it does not generate work for people maintaining the buildbots.
That is fine, but also note that Victor reported a CI issue introduced by the external setuptools package.
I'd really like to build C extensions without downloading an external package.
How often do you actually build extensions without building or installing external packages?
All the time, especially when I'm writing them. I imagine that there's a huge amount of internal company code that discourages use of pip installed packages as well. Or has an air-gapped network in the first place.
Features like C++ support have not been worked on for more than a decade. Are the setuptools maintainers planning to address these issues now?
Considering that we /aren't/ adding anything to distutils today, the chances of this happening in setuptools are pretty much strictly better than in distutils.
Given the time constraints that everyone (rightfully!) has, I'd say the chances are rather low. My point was that features should not be a reason for deprecating distutils. Stefan Krah
On Fri, Sep 4, 2020 at 10:31 AM Stefan Krah <stefan@bytereef.org> wrote:
All the time, especially when I'm writing them. I imagine that there's a huge amount of internal company code that discourages use of pip installed packages as well. Or has an air-gapped network in the first place.
That's wildly hyperbolic; not only will Python retain distutils through 3.11, any "airgapped" build will rest on an existing build that cannot be upgraded, so dependencies are a moot point. -Em
It is not hyperbolic at all. You can get permissions for a certain set of modules (the stdlib), but not for PyPI packages. Of course the upgrade is not via the network, that is beside the point. On Fri, Sep 04, 2020 at 02:56:07PM -0700, Emily Bowman wrote:
On Fri, Sep 4, 2020 at 10:31 AM Stefan Krah <stefan@bytereef.org> wrote:
All the time, especially when I'm writing them. I imagine that there's a huge amount of internal company code that discourages use of pip installed packages as well. Or has an air-gapped network in the first place.
That's wildly hyperbolic; not only will Python retain distutils through 3.11, any "airgapped" build will rest on an existing build that cannot be upgraded, so dependencies are a moot point.
On 9/4/2020 6:00 PM, Stefan Krah wrote:
It is not hyperbolic at all. You can get permissions for a certain set of modules (the stdlib), but not for PyPI packages.
Of course the upgrade is not via the network, that is beside the point. Besides upgrades of existing systems, there are also new installs to consider. To totally remove this functionality is force such new systems to continue using older versions. This is not hypothetical hyperbolic in the least. In the software field I work in, I deal with limited internet connectivity all the time. My single biggest software development problem is software dependencies that assume a full blown internet connection under all circumstances. Even in 2020, that is not always wanted or advisable.
--Edwin Zimmerman
On Fri, Sep 04, 2020 at 02:56:07PM -0700, Emily Bowman wrote:
On Fri, Sep 4, 2020 at 10:31 AM Stefan Krah <stefan@bytereef.org> wrote:
All the time, especially when I'm writing them. I imagine that there's a huge amount of internal company code that discourages use of pip installed packages as well. Or has an air-gapped network in the first place.
That's wildly hyperbolic; not only will Python retain distutils through 3.11, any "airgapped" build will rest on an existing build that cannot be upgraded, so dependencies are a moot point.
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7ONUZMXX... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Sep 4, 2020 at 3:11 PM Stefan Krah <stefan@bytereef.org> wrote:
It is not hyperbolic at all. You can get permissions for a certain set of modules (the stdlib), but not for PyPI packages.
Of course the upgrade is not via the network, that is beside the point.
If you can update to a breaking Python version, but aren't allowed one single point version of an external module, you have a process problem.
On 9/5/2020 3:59 AM, Emily Bowman wrote:
On Fri, Sep 4, 2020 at 3:11 PM Stefan Krah <stefan@bytereef.org <mailto:stefan@bytereef.org>> wrote:
It is not hyperbolic at all. You can get permissions for a certain set of modules (the stdlib), but not for PyPI packages.
Of course the upgrade is not via the network, that is beside the point.
If you can update to a breaking Python version, but aren't allowed one single point version of an external module, you have a process problem. The point remains that these situations exist where it is simply impossible to run 'pip install xyz' due to network restrictions. I know this firsthand because I have written software for enforcing total internet blocks. Pushing Python to a pip-only module will preclude its use in such situations. Again, this is not hypothetical. This is the software world I deal with every day.
On Sat, 5 Sep 2020 07:39:10 -0400 Edwin Zimmerman <edwin@211mainstreet.net> wrote:
On 9/5/2020 3:59 AM, Emily Bowman wrote:
On Fri, Sep 4, 2020 at 3:11 PM Stefan Krah <stefan@bytereef.org <mailto:stefan@bytereef.org>> wrote:
It is not hyperbolic at all. You can get permissions for a certain set of modules (the stdlib), but not for PyPI packages.
Of course the upgrade is not via the network, that is beside the point.
If you can update to a breaking Python version, but aren't allowed one single point version of an external module, you have a process problem. The point remains that these situations exist where it is simply impossible to run 'pip install xyz' due to network restrictions. I know this firsthand because I have written software for enforcing total internet blocks. Pushing Python to a pip-only module will preclude its use in such situations. Again, this is not hypothetical. This is the software world I deal with every day.
Unless you rely on an externally-maintained distribution that provides setuptools for you (such as Anaconda, or others). They are typically meant to deal with that kind of situation. Of course, that does not invalidate the argument that it would be nice if Python itself could still be used without dependencies for that. Regards Antoine.
Thanks everyone for your input so far. Rest assured, I am very aware of air-gapped and limited network systems, as well as corporate policies and processes. Having distutils in the standard library does not help with any of these. Do not forget that pip (and presently, though not permanently, setuptools) are bundled with a recommended CPython distribution through ensurepip. I call out "recommended" because distributors (including those who are transferring CPython into an air-gapped system) can choose to do whatever they like - including leaving out distutils already! As for defining standards for installation, the sysconfig module specifies the paths to install things (though even those can be taken with a grain of salt, as it's really the import system and site module that determine where things should be installed), but we already defer to third parties to do the actual installation (including, yes, those who use distutils to help define their install script). All other packaging and distribution specifications are listed at https://packaging.python.org/specifications/ Also, distutils has never been a "standard" for how to define a package (i.e. write a setup.py), just as argparse is not a "standard" for how to define a CLI. It's always been possible to do it in other ways, and the current standard definition for this is now PEP 517. Since distutils is not compatible with PEP 517, it is explicitly *breaking* the standard. I also wanted to call out this excellent point from Emily:
If you can update to a breaking Python version, but aren't allowed one single point version of an external module, you have a process problem.
This is absolutely the case. CPython, and the core team, cannot take responsibility for bad policies, and (arguably) should not help users work around those policies. It is better to provide useful advice to organisations implementing these policies to help them do things properly (this is a large part of what I get paid to do these days), because otherwise they very quickly just dictate that Python cannot be used at all. Cheers, Steve
On Mon, Sep 07, 2020 at 11:43:41AM +0100, Steve Dower wrote:
Rest assured, I am very aware of air-gapped and limited network systems, as well as corporate policies and processes. Having distutils in the standard library does not help with any of these.
Of course it helps. You can develop extensions without any third party packages or install them. Same situation if you are on mobile Internet or in a cafe without Internet and you want to try something out. Or if you moved and you don't have cable Internet yet. Or if you are in a country where there is no cable Internet. Air-gapped systems were just an illustration of the problem. I did not anticipate that people would take it as the centerpiece of my arguments. Stefan Krah
On 07Sep2020 1424, Stefan Krah wrote:
On Mon, Sep 07, 2020 at 11:43:41AM +0100, Steve Dower wrote:
Rest assured, I am very aware of air-gapped and limited network systems, as well as corporate policies and processes. Having distutils in the standard library does not help with any of these.
Of course it helps. You can develop extensions without any third party packages or install them.
Same situation if you are on mobile Internet or in a cafe without Internet and you want to try something out.
Or if you moved and you don't have cable Internet yet. Or if you are in a country where there is no cable Internet.
Air-gapped systems were just an illustration of the problem. I did not anticipate that people would take it as the centerpiece of my arguments.
Sorry, I didn't mean to imply that - I used "limited network" to capture the rest. I make exactly the same arguments at work from your side for many other situations, so if you'd like I can give you an even more exhaustive list of scenarios where we can't rely on highly available internet :) And yet despite that, I think in this case you're clutching at straws. Sure, you can develop extensions without any third party packages, but you can't develop them with *supported* first party packages. Unless you're offering to take over distutils maintenance? In which case, I'll happily withdraw the PEP :) Cheers, Steve
On Mon, Sep 07, 2020 at 02:40:40PM +0100, Steve Dower wrote:
Air-gapped systems were just an illustration of the problem. I did not anticipate that people would take it as the centerpiece of my arguments.
Sorry, I didn't mean to imply that - I used "limited network" to capture the rest. I make exactly the same arguments at work from your side for many other situations, so if you'd like I can give you an even more exhaustive list of scenarios where we can't rely on highly available internet :)
And yet despite that, I think in this case you're clutching at straws. Sure, you can develop extensions without any third party packages, but you can't develop them with *supported* first party packages.
I'm under the impression that distutils has effectively been frozen for the last decade, except for the substantial improvements you made for the MSVC part. For Unix, no one has addressed e.g. C++ support. The underlying reason has always been that we cannot change anything lest it breaks someone's package. So I have some trouble understanding why we have exercised that kind of restraint (I might have committed a very small C++ patch otherwise) if we can now break everything at once.
Unless you're offering to take over distutils maintenance? In which case, I'll happily withdraw the PEP :)
No, thanks. :) I've looked at the log, most maintenance patches are from a variety of active core devs (this includes, of course, the MSVC patches from you). Will they submit patches to setuptools from now on? Stefan Krah
On 07Sep2020 1602, Stefan Krah wrote:
I'm under the impression that distutils has effectively been frozen for the last decade, except for the substantial improvements you made for the MSVC part.
For Unix, no one has addressed e.g. C++ support. The underlying reason has always been that we cannot change anything lest it breaks someone's package.
So I have some trouble understanding why we have exercised that kind of restraint (I might have committed a very small C++ patch otherwise) if we can now break everything at once.
Others have contributed a range of little fixes over time. Maybe they blur into the background noise of regular maintenance, but it does add up. The main reason we end up needing to fix things is because platforms change things. Since we don't control _when_ platforms will change things, we either do a single controlled break at a planned Python version, or we allow it to break for different users at different times that are not detectable from version number alone. It won't take long for an autoconf-style approach to be necessary for figuring out how to use distutils (though hopefully anyone who sees that as the solution will also find https://packaging.python.org/ and find better solutions ;) )
Unless you're offering to take over distutils maintenance? In which case, I'll happily withdraw the PEP :)
No, thanks. :)
Okay, maybe it is a little bit more than background noise ;)
I've looked at the log, most maintenance patches are from a variety of active core devs (this includes, of course, the MSVC patches from you).
Will they submit patches to setuptools from now on?
If you look at the setuptools history, a variety of contributors have submitted much the same (and often better) fixes there. I expect I likely will as well, inasmuch as they are needed to help Windows users be successful, though I also have my own backend that will likely be where big features that interest me end up ;) It's easier and more satisfying to submit the patches to setuptools, as the release comes out much sooner (and can be installed on earlier versions), and it will only get easier when _all_ the code can be fixed there. Right now, one of the biggest pains there is having to do loads of careful monkeypatching to fix one poor choice in the standard library. Cheers, Steve
On Tue, Sep 8, 2020 at 2:27 AM Steve Dower <steve.dower@python.org> wrote:
On 07Sep2020 1602, Stefan Krah wrote:
I'm under the impression that distutils has effectively been frozen for the last decade, except for the substantial improvements you made for the MSVC part.
For Unix, no one has addressed e.g. C++ support. The underlying reason has always been that we cannot change anything lest it breaks someone's package.
So I have some trouble understanding why we have exercised that kind of restraint (I might have committed a very small C++ patch otherwise) if we can now break everything at once.
Others have contributed a range of little fixes over time. Maybe they blur into the background noise of regular maintenance, but it does add up.
The main reason we end up needing to fix things is because platforms change things. Since we don't control _when_ platforms will change things, we either do a single controlled break at a planned Python version, or we allow it to break for different users at different times that are not detectable from version number alone. It won't take long for an autoconf-style approach to be necessary for figuring out how to use distutils (though hopefully anyone who sees that as the solution will also find https://packaging.python.org/ and find better solutions ;) )
Unless you're offering to take over distutils maintenance? In which case, I'll happily withdraw the PEP :)
No, thanks. :)
Okay, maybe it is a little bit more than background noise ;)
And I think this is the key point. This open source project of ours has some truths about it, including: 1. Someone is using every piece of code somewhere in the world 2. The maintenance of any of it is not free What this PEP is arguing is that the cost/benefit ratio of "people directly using distutils" and "us keeping it afloat" does not work out in our favour enough to warrant us being the primary maintainers. Now I would also say this doesn't mean the setuptools maintainers should be expected to maintain it either; they want the code so they are forking it and are planning to keep it working for their needs. But if anyone else also wants to fork it and keep it running under a different backwards-compatibility policy than setuptools then obviously anyone can.
On 7 Sep 2020, at 14:28, Stefan Krah <stefan@bytereef.org> wrote:
On Mon, Sep 07, 2020 at 11:43:41AM +0100, Steve Dower wrote:
Rest assured, I am very aware of air-gapped and limited network systems, as well as corporate policies and processes. Having distutils in the standard library does not help with any of these.
Of course it helps. You can develop extensions without any third party packages or install them.
But you do not need distutils to write python extensions right? Barry
Same situation if you are on mobile Internet or in a cafe without Internet and you want to try something out.
Or if you moved and you don't have cable Internet yet. Or if you are in a country where there is no cable Internet.
Air-gapped systems were just an illustration of the problem. I did not anticipate that people would take it as the centerpiece of my arguments.
Stefan Krah
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/FNRIPAE4... Code of Conduct: http://python.org/psf/codeofconduct/
On Sat, Sep 5, 2020 at 5:14 PM Edwin Zimmerman <edwin@211mainstreet.net> wrote:
The point remains that these situations exist where it is simply impossible to run 'pip install xyz' due to network restrictions. I know this firsthand because I have written software for enforcing total internet blocks.
I'm not sure I understand how "can't hit the network" => "can't use pip". pip doesn't NEED to hit the network to be functional. The following works without hitting the network. It's even "fairly fast" if you built wheels already. :) $ pip install --no-index --find-links path/to/source/tarballs/and/wheels my-amazing-super-important-package -- Pradyun
As I wrote in https://stackoverflow.com/questions/25337706/setuptools-vs-distutils-why-is-..., distutils has a responsibility that setuptools as a 3rd-party entity cannot fulfill -- to set the core standards for addon modules structure and interface and be guaranteed to be present and compatible with the Python distribution that it comes with. If you are going to remove distutils, something else will need to fulfill these goals. On 04.09.2020 14:28, Steve Dower wrote:
Hi all.
setuptools has recently adopted the entire codebase of the distutils module, so that they will be able to make improvements directly without having to rely on patching the standard library. As a result, we can now move forward with official deprecation (in 3.10) and removal (in 3.12) (noting that the distutils docs already recommend switching to setuptools).
Full text and discussion at https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134
I'm including the original text below, but won't be responding to all discussions here (though I'll periodically check in and skim read it, assuming things don't go way off track).
Also be aware that I already have some minor changes lined up that are not in this text. Refer to the discussion on Discourse if you need to see those.
Cheers, Steve
---
PEP: 632 Title: Deprecate distutils module Author: Steve Dower <steve.dower@python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 03-Sep-2020 Post-History:
Abstract ========
The distutils module [1]_ has for a long time recommended using the setuptools package [2]_ instead. Setuptools has recently integrated a complete copy of distutils and is no longer dependent on the standard library [3]_. Pip has silently replaced distutils with setuptools when building packages for a long time already. It is time to remove it from the (public part of the) standard library.
Motivation ==========
distutils [1]_ is a largely undocumented and unmaintained collection of utilities for packaging and distributing Python packages, including compilation of native extension modules. It defines a configuration format that describes a Python distribution and provides the tools to convert a directory of source code into a source distribution, and some forms of binary distribution. Because of its place in the standard library, many updates can only be released with a major release, and users cannot rely on particular fixes being available.
setuptools [2]_ is a better documented and well maintained enhancement based on distutils. While it provides very similar functionality, it is much better able to support users on earlier Python releases, and can respond to bug reports more quickly. A number of platform-specific enhancements already exist in setuptools that have not been added to distutils, and there is been a long-standing recommendation in the distutils documentation to prefer setuptools.
Historically, setuptools has extended distutils using subclassing and monkeypatching, but has now taken a copy of the underlying code. [3]_ As a result, the second last major dependency on distutils is gone and there is no need to keep it in the standard library.
The final dependency on distutils is CPython itself, which uses it to build the native extension modules in the standard library (except on Windows). Because this is a CPython build-time dependency, it is possible to continue to use distutils for this specific case without it being part of the standard library.
Deprecation and removal will make it obvious that issues should be fixed in the setuptools project, and will reduce a source of bug reports and test maintenance that is unnecessary. It will also help promote the development of alternative build backends, which can now be supported more easily thanks to PEP 517.
Specification =============
In Python 3.10 and 3.11, distutils will be formally marked as deprecated. All known issues will be closed at this time. ``import distutils`` will raise a deprecation warning.
During Python 3.10 and 3.11, uses of distutils within the standard library may change to use alternative APIs.
In Python 3.12, distutils will no longer be installed by ``make install`` or any of the first-party distribution. Third-party redistributors should no longer include distutils in their bundles or repositories.
This PEP makes no specification on migrating the parts of the CPython build process that currently use distutils. Depending on contributions, this migration may occur at any time.
After Python 3.12 is started and when the CPython build process no longer depends on distutils being in the standard library, the entire ``Lib/distutils`` directory and ``Lib/test/test_distutils.py`` file will be removed from the repository.
Other references to distutils will be cleaned up. As of Python 3.9's initial release, the following modules have references in code or comments:
* Lib/ctypes/util.py * Lib/site.py * Lib/sysconfig.py * Lib/_aix_support.py * Lib/_bootsubprocess.py * Lib/_osx_support.py * Modules/_decimal/tests/formathelper.py
As the distutils code is already included in setuptools, there is no need to republish it in any other form. Those who require access to the functionality should use setuptools or an alternative build backend.
Backwards Compatibility =======================
Code that imports distutils will no longer work from Python 3.12.
The suggested migration path is to use the equivalent (though not identical) imports from setuptools.
Compatibility shims already exist in setuptools and pip to transparently migrate old code, however, these are controlled by external projects and are in no way bound by this PEP. Consult their latest documentation for migration advice.
Some projects use alternate sets of shims over distutils, notably, numpy.distutils. [4]_ Where known, these projects have been informed.
Many use custom commands or more narrowly scoped patches, mostly constrained to ``setup.py`` files. As these packages are already subject to setuptools being injected by pip, we expect minimal disruption as a result of distutils being removed.
Reference Implementation ========================
setuptools version 48 includes the complete copy of distutils, and as such is no longer dependent on the standard library's copy. Most implementation issues they have faced are due to the continuing existence of distutils in the standard library, and so removal will improve the stability of their implementation.
There is not yet a reference implementation for the removal of distutils from the standard library, nor is there an implementation for CPython's native module builds without relying on the standard library copy of distutils.
References ==========
.. [1] distutils - Building and installing Python modules (https://docs.python.org/3.9/library/distutils.html)
.. [2] setuptools - PyPI (https://pypi.org/project/setuptools/)
.. [3] setuptools Issue #417 - Adopt distutils (https://github.com/pypa/setuptools/issues/417)
.. [4] Packaging (numpy.distutils) (https://numpy.org/doc/stable/reference/distutils.html)
Copyright =========
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VMFITQ7Z... Code of Conduct: http://python.org/psf/codeofconduct/ -- Regards, Ivan
On Sat, Sep 5, 2020 at 12:37 AM Ivan Pozdeev via Python-Dev < python-dev@python.org> wrote:
As I wrote in https://stackoverflow.com/questions/25337706/setuptools-vs-distutils-why-is-...,
distutils has a responsibility that setuptools as a 3rd-party entity cannot fulfill -- to set the core standards for addon modules structure and interface and be guaranteed to be present and compatible with the Python distribution that it comes with.
If you are going to remove distutils, something else will need to fulfill these goals.
Obviously, the "something else" is setuptools, which has filled that niche for well over a decade now, and shows no signs of going away anytime soon. Maybe someday there might be a PEP 517 challenger, but for now, setuptools isn't going anywhere even if distutils goes away.
Then, as Victor said, it will have to be bundled into Python's codebase. On 05.09.2020 11:06, Emily Bowman wrote:
On Sat, Sep 5, 2020 at 12:37 AM Ivan Pozdeev via Python-Dev <python-dev@python.org <mailto:python-dev@python.org>> wrote:
As I wrote in https://stackoverflow.com/questions/25337706/setuptools-vs-distutils-why-is-..., distutils has a responsibility that setuptools as a 3rd-party entity cannot fulfill -- to set the core standards for addon modules structure and interface and be guaranteed to be present and compatible with the Python distribution that it comes with.
If you are going to remove distutils, something else will need to fulfill these goals.
Obviously, the "something else" is setuptools, which has filled that niche for well over a decade now, and shows no signs of going away anytime soon. Maybe someday there might be a PEP 517 challenger, but for now, setuptools isn't going anywhere even if distutils goes away. -- Regards, Ivan
On 9/4/20 1:28 PM, Steve Dower wrote:
Hi all.
setuptools has recently adopted the entire codebase of the distutils module, so that they will be able to make improvements directly without having to rely on patching the standard library. As a result, we can now move forward with official deprecation (in 3.10) and removal (in 3.12) (noting that the distutils docs already recommend switching to setuptools).
it would be nice to already announce that with the 3.9 release. At the 2018 Language summit, I had a lightning talk to report about the experience splitting out distutils into a separate binary package, showing some unexpected usages: Unexpected / Creative usages: - distutils.version Used “everywhere” ... - distutils.spawn: find_executable Replace with shutil.which - distutils.util: strtobool Rewrite, no equivalent in the stdlib? - distutils.sysconfig: Mostly replaced by sysconfig It really would be nice to have recommended replacements, especially for the version stuff (packaging?) Matthias
On Fri, Sep 11, 2020 at 4:25 AM Matthias Klose <doko@ubuntu.com> wrote:
On 9/4/20 1:28 PM, Steve Dower wrote:
Hi all.
setuptools has recently adopted the entire codebase of the distutils module, so that they will be able to make improvements directly without having to rely on patching the standard library. As a result, we can now move forward with official deprecation (in 3.10) and removal (in 3.12) (noting that the distutils docs already recommend switching to setuptools).
it would be nice to already announce that with the 3.9 release.
At the 2018 Language summit, I had a lightning talk to report about the experience splitting out distutils into a separate binary package, showing some unexpected usages:
Unexpected / Creative usages:
- distutils.version Used “everywhere” ...
- distutils.spawn: find_executable Replace with shutil.which
- distutils.util: strtobool Rewrite, no equivalent in the stdlib?
- distutils.sysconfig: Mostly replaced by sysconfig
It really would be nice to have recommended replacements, especially for the version stuff (packaging?)
Version support has been in 'packaging' for years: https://packaging.pypa.io/en/latest/version/. -Brett
On 9/11/2020 12:24 PM, Matthias Klose wrote:
On 9/4/20 1:28 PM, Steve Dower wrote:
Hi all.
setuptools has recently adopted the entire codebase of the distutils module, so that they will be able to make improvements directly without having to rely on patching the standard library. As a result, we can now move forward with official deprecation (in 3.10) and removal (in 3.12) (noting that the distutils docs already recommend switching to setuptools).
it would be nice to already announce that with the 3.9 release.
I think the announcement will have an immediate effect regardless of what versions it's tied to.
At the 2018 Language summit, I had a lightning talk to report about the experience splitting out distutils into a separate binary package, showing some unexpected usages:
Unexpected / Creative usages:
- distutils.version Used “everywhere” ...
- distutils.spawn: find_executable Replace with shutil.which
- distutils.util: strtobool Rewrite, no equivalent in the stdlib?
- distutils.sysconfig: Mostly replaced by sysconfig
(Aside - I've put out a call on Discourse for people to contribute the parts of distutils that are used but wouldn't be suitable for using setuptools instead. These are good, but don't seem drastic enough to not deprecate the module. Thoughts?)
It really would be nice to have recommended replacements, especially for the version stuff (packaging?)
Considering that the PEP won't be updated after acceptance, and all of these recommendations would be to third-party packages (except possibly strtobool), how relevant would you predict these to be in three years time? Setuptools has already been the official recommendation for the whole module for a long time, and their page at https://setuptools.readthedocs.io/en/latest/distutils-legacy.html already lists a few replacements (and I'm sure they'll accept contributions :) ). So I'd prefer to point towards that as the most up-to-date source of recommendations. Thoughts? Cheers, Steve
participants (15)
-
Antoine Pitrou -
Barry -
Brett Cannon -
Edwin Zimmerman -
Emily Bowman -
Fred Drake -
Ivan Pozdeev -
Jim J. Jewett -
Matthias Klose -
Paul Ganssle -
Paul Moore -
Pradyun Gedam -
Stefan Krah -
Steve Dower -
Victor Stinner