Migration plan for the distutils removal in Python 3.12
Hi, Two years ago, PEP 632 "Deprecate distutils module" was accepted: the distutils package was deprecated in Python 3.10 and scheduled for removal in Python 3.12. Questions. * Is the Python ecosystem ready for the distutils removal? How many projects are expected to be broken by this removal? * Is setuptools a drop-in replacement of distutils for most cases? Are there tools and documentation explaining how to replace distutils with setuptools? * Is there a tool to migrate from setup.py (distutils) to pyproject.toml (setuptools)? The dephell project can convert a setup.py script to pyproject.toml using poetry as the build system. * Can we simply suggest installing setuptools to users who still use "import distutils"? setuptools now provides the a "distutils" package. * Should we keep distutils in Python stdlib a few more Python releases if the removal causes too many practical issues? A code search in top 5000 PyPI projects (at 2022-01-26) using the regex '(import|from) distutils' matchs 5,567 lines in 1,229 projects. Some statistics: * 851 projects (1,372 lines) import distutils in setup.py * 233 projects (700 lines) use distutils.version * 92 projects (205 lines) use distutils.util * 1,018 lines are type annotations (.pyi files) I failed to group the remaining 2,272 lines. Converting 851 projects from setup.py with distutils to pyproject.toml is going to take a few months if not years. Python 3.12 will likely be released in October 2023 (in year and a half). Since setuptools 60 (December 2021), setuptools installs a distutils-precedence.pth file to override the stdlib module with its local distutils copy by default. The SETUPTOOLS_USE_DISTUTILS=stdlib environment variable can be used to explicitly request the stdlib flavor. If I understood correct, distutils is now maintained in setuptools, rather than in the Python stdlib. Python "make install" installs an up to date setuptools and so indirectly its distutils-precedence.pth file which makes the distutils local copy of setuptools available. But many Linux distributions splits setuptools and pip from the "python" package. Right now, Python still uses distutils internally for multiple use cases. I propose to start with renaming the distutils package to _distutils in the stdlib: * https://github.com/python/cpython/issues/92584 * https://github.com/python/cpython/pull/92585 Right now, the change is blocked by pip which still imports distutils (even if it prefers sysconfig in practice on Python 3.10 and newer). I reported the issue to pip, it should be easy to fix. Once Python will no longer depend on _distutils, it will be possible to fully remove it. By the way, Fedora 35 now longer provides automatically setuptools when building a Python package. A dependency to setuptools must now be explicit, since they are projects which don't use setuptools nor distutils, but other build systems. https://fedoraproject.org/wiki/Changes/Reduce_dependencies_on_python3-setupt... Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Tue, 2022-05-10 at 09:53 +0200, Victor Stinner wrote:
Hi,
Two years ago, PEP 632 "Deprecate distutils module" was accepted: the distutils package was deprecated in Python 3.10 and scheduled for removal in Python 3.12. Questions.
* Is the Python ecosystem ready for the distutils removal? How many projects are expected to be broken by this removal?
In Gentoo, I have decided to handle migration from `setup.py install` to PEP517 backends and distutils replacement simultaneously. That is, existing packages that still use the legacy codepath have =stdlib distutils forced, and =local distutils are used when they are updated to use PEP517 build. So far we're nearing updating 40% of Python packages in Gentoo. From my porting experience, I can't think of any major issue caused by replacing stdlib distutils with the setuptools version. IIRC the only problem I recall was that the "local" version installs .egg-info as a directory (like setuptools does) while "stdlib" version installs it as a file. This could mean trouble when the same version of the package is rebuilt with the other distutils version but I don't know how that impacts other package managers. So far, much more trouble is caused by PEP517 backends and wheels, though admittedly this is primarily a problem with setup.py allowing very deep customization and people using it to do pretty much anything. Therefore, from Gentoo's perspective, there should be no major problem with removing distutils from stdlib in Python 3.12. Admittedly, distributions are in the more convenient position here since we can easily patch packages should we find any breakage, while users of pypi are generally stuck with the published versions. -- Best regards, Michał Górny
On 10/05/2022 09.53, Victor Stinner wrote:
Hi,
Two years ago, PEP 632 "Deprecate distutils module" was accepted: the distutils package was deprecated in Python 3.10 and scheduled for removal in Python 3.12. Questions.
* Is the Python ecosystem ready for the distutils removal? How many projects are expected to be broken by this removal?
* Is setuptools a drop-in replacement of distutils for most cases? Are there tools and documentation explaining how to replace distutils with setuptools?
* Is there a tool to migrate from setup.py (distutils) to pyproject.toml (setuptools)? The dephell project can convert a setup.py script to pyproject.toml using poetry as the build system.
* Can we simply suggest installing setuptools to users who still use "import distutils"? setuptools now provides the a "distutils" package.
* Should we keep distutils in Python stdlib a few more Python releases if the removal causes too many practical issues?
Setuptools now provides _distutils_hack meta importer that injects a facade distutils package. "from distutils import setup" will keep working with modern setuptools.
Right now, Python still uses distutils internally for multiple use cases. I propose to start with renaming the distutils package to _distutils in the stdlib:
* https://github.com/python/cpython/issues/92584 * https://github.com/python/cpython/pull/92585
Erlend and I got most extension modules ported to autoconf and Modules/Setup.stdlib. The remaining modules should be done in a couple of weeks. I recommend that we do not rename distutils and instead remove it entirely. Christian
On 10/05/2022 09.53, Victor Stinner wrote:
Hi,
Two years ago, PEP 632 "Deprecate distutils module" was accepted: the distutils package was deprecated in Python 3.10 and scheduled for removal in Python 3.12. Questions.
* Is the Python ecosystem ready for the distutils removal? How many projects are expected to be broken by this removal?
* Is setuptools a drop-in replacement of distutils for most cases? Are there tools and documentation explaining how to replace distutils with setuptools?
* Is there a tool to migrate from setup.py (distutils) to pyproject.toml (setuptools)? The dephell project can convert a setup.py script to pyproject.toml using poetry as the build system.
* Can we simply suggest installing setuptools to users who still use "import distutils"? setuptools now provides the a "distutils" package.
* Should we keep distutils in Python stdlib a few more Python releases if the removal causes too many practical issues?
Setuptools now provides _distutils_hack meta importer that injects a facade distutils package. "from distutils import setup" will keep working with modern setuptools.
Right now, Python still uses distutils internally for multiple use cases. I propose to start with renaming the distutils package to _distutils in the stdlib:
* https://github.com/python/cpython/issues/92584 * https://github.com/python/cpython/pull/92585
Erlend and I got most extension modules ported to autoconf and Modules/Setup.stdlib. The remaining modules should be done in a couple of weeks. I recommend that we do not rename distutils and instead remove it entirely. Christian
On Tue, May 10, 2022 at 11:28 AM Christian Heimes <christian@python.org> wrote:
Right now, Python still uses distutils internally for multiple use cases. I propose to start with renaming the distutils package to _distutils in the stdlib:
* https://github.com/python/cpython/issues/92584 * https://github.com/python/cpython/pull/92585
Erlend and I got most extension modules ported to autoconf and Modules/Setup.stdlib. The remaining modules should be done in a couple of weeks. I recommend that we do not rename distutils and instead remove it entirely.
test_peg_generator and test_cppext build C extensions with distutils. Until these tests are modified to use something else, we still need to keep distutils. So I propose to rename it to _distutils to remove it from the stdlib. Maybe these tests can use ensurepip to install setuptools which provides distutils. There is also the c-analyzer tool which uses distutils to run a C preprocessor. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 10/05/2022 13.18, Victor Stinner wrote:
On Tue, May 10, 2022 at 11:28 AM Christian Heimes <christian@python.org> wrote:
Right now, Python still uses distutils internally for multiple use cases. I propose to start with renaming the distutils package to _distutils in the stdlib:
* https://github.com/python/cpython/issues/92584 * https://github.com/python/cpython/pull/92585
Erlend and I got most extension modules ported to autoconf and Modules/Setup.stdlib. The remaining modules should be done in a couple of weeks. I recommend that we do not rename distutils and instead remove it entirely.
test_peg_generator and test_cppext build C extensions with distutils. Until these tests are modified to use something else, we still need to keep distutils. So I propose to rename it to _distutils to remove it from the stdlib. Maybe these tests can use ensurepip to install setuptools which provides distutils.
There is also the c-analyzer tool which uses distutils to run a C preprocessor.
We can easily take care of test_cppext and add the build step to Makefile. How does test_peg_generator depend on distutils? I don't see an import of distutils in the directory. I would prefer to fix and remove all imports of distutils before we resort to renaming the package. Please give us time until alpha 1. There is no need to rush it *now*. Christian
On Tue, May 10, 2022 at 9:34 AM Christian Heimes <christian@python.org> wrote:
We can easily take care of test_cppext and add the build step to Makefile. How does test_peg_generator depend on distutils? I don't see an import of distutils in the directory.
You need to look at the files in the Tools directory. That is where c-analyzer and peg_generator live. The peg_generator tests depend on Tools/peg_generator/pegen/build.py which uses distutils to build multiple extensions to test the grammar generator. -- Jeremy Kloth
On Tue, May 10, 2022 at 5:33 PM Christian Heimes <christian@python.org> wrote:
We can easily take care of test_cppext and add the build step to Makefile.
What matters in test_cppext is that using the Python C API in C++ does not emit compiler warnings, so it uses -Werror. Since this test is very new, I would prefer to not block the Python build just because of that. Moreover, I'm not sure that I want to require a C++ compiler to build CPython (written in C). But yeah, something can be done in Makefile :-)
How does test_peg_generator depend on distutils? I don't see an import of distutils in the directory.
See Tools/peg_generator/pegen/build.py file. The test builds C extensions and then execute them.
I would prefer to fix and remove all imports of distutils before we resort to renaming the package. Please give us time until alpha 1. There is no need to rush it *now*.
Sure, there is no need to rush it now. But I prefer to discuss the change as soon as possible (in the 3.12 dev cycle), to make sure that we are aware of all issues, and have more time to fix them. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 5/10/2022 4:33 PM, Christian Heimes wrote:
On 10/05/2022 13.18, Victor Stinner wrote:
test_peg_generator and test_cppext build C extensions with distutils. Until these tests are modified to use something else, we still need to keep distutils. So I propose to rename it to _distutils to remove it from the stdlib. Maybe these tests can use ensurepip to install setuptools which provides distutils.
There is also the c-analyzer tool which uses distutils to run a C preprocessor.
We can easily take care of test_cppext and add the build step to Makefile. How does test_peg_generator depend on distutils? I don't see an import of distutils in the directory.
I would prefer to fix and remove all imports of distutils before we resort to renaming the package. Please give us time until alpha 1. There is no need to rush it *now*.
I agree. The internal tools that use it are all in our Tools directory, so we can move distutils there and explicitly add an import path to locate it. No need to keep it in the stdlib (Lib/) at all. Migrating to Makefile builds is probably better long-term, but not as important as moving distutils out from the stdlib so that setuptools can rely on their copy being the "main" one. Cheers, Steve
Shouldn't we wean our internal tools off this obsolete version of distutils too, rather than just move it around? On Tue, May 10, 2022 at 9:26 AM Steve Dower <steve.dower@python.org> wrote:
On 5/10/2022 4:33 PM, Christian Heimes wrote:
On 10/05/2022 13.18, Victor Stinner wrote:
test_peg_generator and test_cppext build C extensions with distutils. Until these tests are modified to use something else, we still need to keep distutils. So I propose to rename it to _distutils to remove it from the stdlib. Maybe these tests can use ensurepip to install setuptools which provides distutils.
There is also the c-analyzer tool which uses distutils to run a C preprocessor.
We can easily take care of test_cppext and add the build step to Makefile. How does test_peg_generator depend on distutils? I don't see an import of distutils in the directory.
I would prefer to fix and remove all imports of distutils before we resort to renaming the package. Please give us time until alpha 1. There is no need to rush it *now*.
I agree. The internal tools that use it are all in our Tools directory, so we can move distutils there and explicitly add an import path to locate it. No need to keep it in the stdlib (Lib/) at all.
Migrating to Makefile builds is probably better long-term, but not as important as moving distutils out from the stdlib so that setuptools can rely on their copy being the "main" one.
Cheers, Steve
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LGF4BJMN... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Tue, May 10, 2022 at 10:23 AM Guido van Rossum <guido@python.org> wrote:
Shouldn't we wean our internal tools off this obsolete version of distutils too, rather than just move it around?
I think so. We technically have a year to get this done, so if we can't do that then I'm afraid we have much bigger issues with the project's maintenance. -Brett
On Tue, May 10, 2022 at 9:26 AM Steve Dower <steve.dower@python.org> wrote:
On 5/10/2022 4:33 PM, Christian Heimes wrote:
On 10/05/2022 13.18, Victor Stinner wrote:
test_peg_generator and test_cppext build C extensions with distutils. Until these tests are modified to use something else, we still need to keep distutils. So I propose to rename it to _distutils to remove it from the stdlib. Maybe these tests can use ensurepip to install setuptools which provides distutils.
There is also the c-analyzer tool which uses distutils to run a C preprocessor.
We can easily take care of test_cppext and add the build step to Makefile. How does test_peg_generator depend on distutils? I don't see an import of distutils in the directory.
I would prefer to fix and remove all imports of distutils before we resort to renaming the package. Please give us time until alpha 1. There is no need to rush it *now*.
I agree. The internal tools that use it are all in our Tools directory, so we can move distutils there and explicitly add an import path to locate it. No need to keep it in the stdlib (Lib/) at all.
Migrating to Makefile builds is probably better long-term, but not as important as moving distutils out from the stdlib so that setuptools can rely on their copy being the "main" one.
Cheers, Steve
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/LGF4BJMN... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...> _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L3JRME4C... Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, May 10, 2022 at 7:18 PM Guido van Rossum <guido@python.org> wrote:
Shouldn't we wean our internal tools off this obsolete version of distutils too, rather than just move it around?
Here is a first PR to build the test C++ extension with setuptools in test_cppext: https://github.com/python/cpython/pull/92639 It implements what I proposed in the issue. It runs the test in a venv. Python stdlib (ensurepip) includes a setuptools wheel package and so is able to install it without adding any new external dependency. If this approach works, we can use the same for test_peg_generator. Victor
On Tue, May 10, 2022 at 6:16 PM Steve Dower <steve.dower@python.org> wrote:
I agree. The internal tools that use it are all in our Tools directory, so we can move distutils there and explicitly add an import path to locate it. No need to keep it in the stdlib (Lib/) at all.
Migrating to Makefile builds is probably better long-term, but not as important as moving distutils out from the stdlib so that setuptools can rely on their copy being the "main" one.
With my current PR, Lib/_distutils/ is not installed by "make install". Moving it to Tools/ would also work. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
Just a note: The huge amount of work/progress that has gone into modernizing the building/packaging tools is great, but to be honest, C extensions have been somewhat neglected. e.g. I still need to use old fashioned setup.py to build some of my projects (C and Cython extensions) This may not be relevant to this discussion, as I think setuptools does still support C extensions as well as it has for years -- All of my recent (last ten years) projects use it. But it makes me nervous when I see suggestions to keep distutils in the stdlib: if the stdlib finds it still needs distutils, then that means that there may be others in the community that need it too. So if you can't entirely drop distutils in the stdlib, I don't think we can count on the rest of the community being able to drop it as well. Thanks, -CHB On Tue, May 10, 2022 at 1:29 PM Victor Stinner <vstinner@python.org> wrote:
On Tue, May 10, 2022 at 6:16 PM Steve Dower <steve.dower@python.org> wrote:
I agree. The internal tools that use it are all in our Tools directory, so we can move distutils there and explicitly add an import path to locate it. No need to keep it in the stdlib (Lib/) at all.
Migrating to Makefile builds is probably better long-term, but not as important as moving distutils out from the stdlib so that setuptools can rely on their copy being the "main" one.
With my current PR, Lib/_distutils/ is not installed by "make install". Moving it to Tools/ would also work.
Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/T43OZ4LW... Code of Conduct: http://python.org/psf/codeofconduct/
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 5/10/2022 9:38 PM, Christopher Barker wrote:
So if you can't entirely drop distutils in the stdlib, I don't think we can count on the rest of the community being able to drop it as well.
The only reason distutils is used to build Python is for self-bootstrapping. setuptools would have to avoid any module that may potentially be built as an extension module if it were to be used at that stage of build, which is unfair (and this use is also likely to become Makefile for Linux and macOS, just as it uses static build scripts on Windows already). The other uses are tests or tools. There's nothing wrong with setuptools's functionality, merely the fact that it isn't part of the CPython repository and we don't like relying on third-party repositories for a CPython build/test. Cheers, Steve
If we're not shipping it, we should just move it out of the way. Otherwise, there are a few places in the various Windows packages that will need it to be explicitly excluded. We should already have similar exclusions for the test suite and a few other modules, but they're a bit scattered around PC, PCbuild and Tools/msi. Easier to just remove it from Lib. Cheers, Steve On 5/10/2022 9:24 PM, Victor Stinner wrote:
On Tue, May 10, 2022 at 6:16 PM Steve Dower <steve.dower@python.org> wrote:
I agree. The internal tools that use it are all in our Tools directory, so we can move distutils there and explicitly add an import path to locate it. No need to keep it in the stdlib (Lib/) at all.
Migrating to Makefile builds is probably better long-term, but not as important as moving distutils out from the stdlib so that setuptools can rely on their copy being the "main" one.
With my current PR, Lib/_distutils/ is not installed by "make install". Moving it to Tools/ would also work.
Victor
On Tue, May 10, 2022 at 12:59 AM Victor Stinner <vstinner@python.org> wrote:
Hi,
Two years ago, PEP 632 "Deprecate distutils module" was accepted: the distutils package was deprecated in Python 3.10 and scheduled for removal in Python 3.12. Questions.
* Is the Python ecosystem ready for the distutils removal? How many projects are expected to be broken by this removal?
* Is setuptools a drop-in replacement of distutils for most cases? Are there tools and documentation explaining how to replace distutils with setuptools?
Change `import distutils` with `import setuptools`. 😉 But more thorough guidance can be found at https://setuptools.pypa.io/en/latest/deprecated/distutils-legacy.html?highli... .
* Is there a tool to migrate from setup.py (distutils) to pyproject.toml (setuptools)? The dephell project can convert a setup.py script to pyproject.toml using poetry as the build system.
You don't have to migrate a setuptools project to pyproject.toml to keep functioning. I would strongly encourage people at least specify a build-system table in pyproject.toml as it's easy to do, but pip doesn't require even this for setuptools projects: ```toml [build-system]requires = ["setuptools"]build-backend = "setuptools.build_meta" ``` But the rest of the project's metadata can stay in setup.py or setup.cfg if they want. But there is experimental support to fully support PEP 621 and specifying metadata in pyproject.toml: https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html .
* Can we simply suggest installing setuptools to users who still use "import distutils"? setuptools now provides the a "distutils" package.
If they are unwilling to update, yes.
* Should we keep distutils in Python stdlib a few more Python releases if the removal causes too many practical issues?
I don't think so. -Brett
A code search in top 5000 PyPI projects (at 2022-01-26) using the regex '(import|from) distutils' matchs 5,567 lines in 1,229 projects. Some statistics:
* 851 projects (1,372 lines) import distutils in setup.py * 233 projects (700 lines) use distutils.version * 92 projects (205 lines) use distutils.util * 1,018 lines are type annotations (.pyi files)
I failed to group the remaining 2,272 lines.
Converting 851 projects from setup.py with distutils to pyproject.toml is going to take a few months if not years. Python 3.12 will likely be released in October 2023 (in year and a half).
Since setuptools 60 (December 2021), setuptools installs a distutils-precedence.pth file to override the stdlib module with its local distutils copy by default. The SETUPTOOLS_USE_DISTUTILS=stdlib environment variable can be used to explicitly request the stdlib flavor. If I understood correct, distutils is now maintained in setuptools, rather than in the Python stdlib.
Python "make install" installs an up to date setuptools and so indirectly its distutils-precedence.pth file which makes the distutils local copy of setuptools available. But many Linux distributions splits setuptools and pip from the "python" package.
Right now, Python still uses distutils internally for multiple use cases. I propose to start with renaming the distutils package to _distutils in the stdlib:
* https://github.com/python/cpython/issues/92584 * https://github.com/python/cpython/pull/92585
Right now, the change is blocked by pip which still imports distutils (even if it prefers sysconfig in practice on Python 3.10 and newer). I reported the issue to pip, it should be easy to fix.
Once Python will no longer depend on _distutils, it will be possible to fully remove it.
By the way, Fedora 35 now longer provides automatically setuptools when building a Python package. A dependency to setuptools must now be explicit, since they are projects which don't use setuptools nor distutils, but other build systems.
https://fedoraproject.org/wiki/Changes/Reduce_dependencies_on_python3-setupt...
Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XFZJ43VS... Code of Conduct: http://python.org/psf/codeofconduct/
participants (9)
-
Brett Cannon -
Christian Heimes -
Christian Heimes -
Christopher Barker -
Guido van Rossum -
Jeremy Kloth -
Michał Górny -
Steve Dower -
Victor Stinner