deprecating numpy.distutils
Hi all, Now that 1.22.0 is out the door, I think it's time to deprecate `numpy.distutils` completely. A brief summary for context: - `distutils` was deprecated in Python 3.10, and will be removed in Python 3.12 (Oct 2023 release date) - `setuptools` has re-enabled its vendored copy of `distutils` in its 60.0 release. It's incompatible with `numpy.distutils` and many more breaking changes are to be expected. - Rebasing `numpy.distutils` on top of `setuptools` will be a moving target. More importantly, `numpy.distutils` still has serious and hard to fix bugs and design issues. For example, we cannot build SciPy in parallel due to race conditions in the Fortran support code, and there's a very elusive bug where building an extension module which links in a static Fortran-only library is underlinked and breaks at runtime in dev environments. My initial plan was to contribute Fortran support and other `numpy.distutils` features to `setuptools`, but I've personally changed my mind on that and don't plan to do that work. I think `setuptools` is moving too slow for this to be a productive exercise, and the design of distutils is fundamentally unfixable. So it's a much better investment of time to help projects migrate away from `numpy.distutils` (and hence from `setuptools`). The tracking issue for this is https://github.com/numpy/numpy/issues/18588. Please see the comments and linked issue/thread for more details. My proposal is: 1. deprecate `numpy.distutils` now in main. 2. keep it around for another 2 years after the 1.23.0 release. On >=3.12 it will raise an informative error when someone tries to import it. 3. Only setuptools < 60.0 is supported. This is the status as of today - unless there's an issue with Python 3.11 support, I don't see a point in spending time on keeping up with new setuptools releases, a pin is just fine. 4. Write a docs page with migration info on how to deal with the deprecation, and update it before the 1.23.0 release. Note that the setuptools maintainers have said that they are happy to consider new features if people want to integrate them. The most popular/useful features for `numpy.distutils` are (1) Fortran support, and (2) nested setup.py support. The former is probably not a good idea to integrate, the latter may be. There aren't many packages who need BLAS/LAPACK support anymore, scikit-learn/statsmodels/etc. switched to the Cython/C API exposed by SciPy, rather than linking to BLAS/LAPACK directly. For NumPy itself, we should move to Meson after the SciPy 1.9.0 release has gone out and ironed out all the remaining issues there. We can revisit that in 6 months or so. Thoughts? Cheers, Ralf
4. Write a docs page with migration info on how to deal with the deprecation, and update it before the 1.23.0 release.
i think this will be extremely important. Just to give you the current POV of debian projects using numpy.distutils: - https://codesearch.debian.net/search?q=numpy.distutils+-package%3Anumpy&literal=0&perpkg=1 (there are 74 of them) - https://codesearch.debian.net/search?q=from+numpy+import+.*distutils&literal=0 (only one "from numpy import distutils") hope this could help you come up with a smooth deprecation process :) thanks, -- Sandro "morph" Tosi My website: http://sandrotosi.me/ Me at Debian: http://wiki.debian.org/SandroTosi Twitter: https://twitter.com/sandrotosi
On Sun, Jan 16, 2022 at 10:40 PM Sandro Tosi <sandro.tosi@gmail.com> wrote:
4. Write a docs page with migration info on how to deal with the deprecation, and update it before the 1.23.0 release.
i think this will be extremely important. Just to give you the current POV of debian projects using numpy.distutils:
- https://codesearch.debian.net/search?q=numpy.distutils+-package%3Anumpy&literal=0&perpkg=1 (there are 74 of them) - https://codesearch.debian.net/search?q=from+numpy+import+.*distutils&literal=0 (only one "from numpy import distutils")
hope this could help you come up with a smooth deprecation process :)
Thanks Sandro, that is helpful. I had a look at all those 15 pages of search results: PyPy: just a few patches for numpy <1.11.2 which can be dropped Silx: uses Configuration (i.e., nested setup.py's) Electrum: just a few patches for numpy <1.11.2 which can be dropped Setuptools: just a few patches for numpy <1.11.2 which can be dropped python-hdf4: nothing special, can be switched to setuptools pybigwig: uses `get_info('npymath')` to (optionally) build against numpy - this may need to be added to setuptools python-pot: uses private numpy.distutils compiler support; can be switched to setuptools or vendored petsc4py: small setup.py, but uses Fortran support and Configuration - should probably switch build system basemap: uses Extension, can just switch to setuptools openexr: just queries numpy include dirs in an incorrect fashion, should use `numpy.get_include` rocketcea: nothing special, can just switch to setuptools brian: no usage, just a string `numpy.distutils` in docs libvigraimpex: just queries numpy include dirs in an incorrect fashion, should use `numpy.get_include` gpaw: some weird monkeypatching in shell scripts of numpy's BLAS/LAPACK support, can probably just be dropped pyferret: uses Extension, can just switch to setuptools adolc: just queries numpy include dirs in an incorrect fashion, should use `numpy.get_include` theano: uses BLAS support, they should probably vendor `system_info.py` (or switch build systems) scikit-image: uses Configuration scikit-ffm: uses Configuration, but only two setup.py files - easy to consolidate in a single setup.py and switch to setuptools pywavelets: no longer uses numpy.distutils pybind11: just a code comment, no numpy.distutils usage sfepy: uses Configuration nibabel: only one usage in a test case, just drop it versioneer: just a mention in the docs cypari: just a patch for numpy < 1.11.2, can be dropped blender: just a patch that can be dropped scikit-learn: uses Configuration and build_ext modification for parallel compilation nipype: minor usage of Configuration, can be dropped guiqwt: uses Extension but no longer has Fortran code, so can switch to setuptools The summary of this is: if Configuration is added to setuptools, 90% of packages are fine switching to setuptools. Scikit-learn is planning to follow SciPy and move to Meson. Theano is the only real consumer of numpy.distutils BLAS/LAPACK support, which they should vendor if they don't want to switch build systems. There are no other serious issues here. I'm sure there are some more packages out there relying on Fortran support in particular, they are best off moving to Meson or CMake. Cheers, Ralf
On Mon, 17 Jan 2022 at 10:01, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sun, Jan 16, 2022 at 10:40 PM Sandro Tosi <sandro.tosi@gmail.com> wrote:
4. Write a docs page with migration info on how to deal with the deprecation, and update it before the 1.23.0 release.
i think this will be extremely important. Just to give you the current POV of debian projects using numpy.distutils:
- https://codesearch.debian.net/search?q=numpy.distutils+-package%3Anumpy&literal=0&perpkg=1 (there are 74 of them) - https://codesearch.debian.net/search?q=from+numpy+import+.*distutils&literal=0 (only one "from numpy import distutils")
hope this could help you come up with a smooth deprecation process :)
Thanks Sandro, that is helpful. I had a look at all those 15 pages of search results:
...
Theano is the only real consumer of numpy.distutils BLAS/LAPACK support, which they should vendor if they don't want to switch build systems.
Note that Theano is no longer maintained as seen at the top of the README: https://github.com/Theano/Theano Forked development continues under the name Aesara: https://github.com/aesara-devs/aesara Not sure if Aesara differs in respect of using numpy.distutils but Theano is 2 years out of date in comparison. -- Oscar
On Mon, Jan 17, 2022 at 11:27 AM Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
On Mon, 17 Jan 2022 at 10:01, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sun, Jan 16, 2022 at 10:40 PM Sandro Tosi <sandro.tosi@gmail.com> wrote:
4. Write a docs page with migration info on how to deal with the deprecation, and update it before the 1.23.0 release.
i think this will be extremely important. Just to give you the current POV of debian projects using numpy.distutils:
- https://codesearch.debian.net/search?q=numpy.distutils+-package%3Anumpy&literal=0&perpkg=1 (there are 74 of them) - https://codesearch.debian.net/search?q=from+numpy+import+.*distutils&literal=0 (only one "from numpy import distutils")
hope this could help you come up with a smooth deprecation process :)
Thanks Sandro, that is helpful. I had a look at all those 15 pages of search results:
...
Theano is the only real consumer of numpy.distutils BLAS/LAPACK support, which they should vendor if they don't want to switch build systems.
Note that Theano is no longer maintained as seen at the top of the README: https://github.com/Theano/Theano
Forked development continues under the name Aesara: https://github.com/aesara-devs/aesara
Not sure if Aesara differs in respect of using numpy.distutils but Theano is 2 years out of date in comparison.
Thanks for pointing that out Oscar. Still the same situation, one nontrivial use to query for BLAS config: https://github.com/aesara-devs/aesara/blob/4468ba601f96d82a387ab0d8ab09bdedb... Cheers, Ralf
Hi, I should mention many of our software projects (fabio, pyFAI, ...) rely on silx for the nested setup.py part, and not all are mentionned in this list. In the absence of alternative solution, we limit the version of setup.py for now but we are willing to help in finding a solution (our complete build stack would collapses). Well the parallel build feature is a very much appreciated because the serial compilation of our project take dozens of minutes and our computers are HPC nodes with plenty of cores... I find funny the way of first deprecate ... then look if any alternative solution exists. If I understand properly, your idea is to port the nested setup.py part from numpy.distutils to setuptools ? Are people of setuptools already in the loop ? Cheers, Jerome On Mon, 17 Jan 2022 10:57:40 +0100 Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sun, Jan 16, 2022 at 10:40 PM Sandro Tosi <sandro.tosi@gmail.com> wrote:
4. Write a docs page with migration info on how to deal with the deprecation, and update it before the 1.23.0 release.
i think this will be extremely important. Just to give you the current POV of debian projects using numpy.distutils:
- https://codesearch.debian.net/search?q=numpy.distutils+-package%3Anumpy&literal=0&perpkg=1 (there are 74 of them) - https://codesearch.debian.net/search?q=from+numpy+import+.*distutils&literal=0 (only one "from numpy import distutils")
hope this could help you come up with a smooth deprecation process :)
Thanks Sandro, that is helpful. I had a look at all those 15 pages of search results:
PyPy: just a few patches for numpy <1.11.2 which can be dropped Silx: uses Configuration (i.e., nested setup.py's) Electrum: just a few patches for numpy <1.11.2 which can be dropped Setuptools: just a few patches for numpy <1.11.2 which can be dropped python-hdf4: nothing special, can be switched to setuptools pybigwig: uses `get_info('npymath')` to (optionally) build against numpy - this may need to be added to setuptools python-pot: uses private numpy.distutils compiler support; can be switched to setuptools or vendored petsc4py: small setup.py, but uses Fortran support and Configuration - should probably switch build system basemap: uses Extension, can just switch to setuptools openexr: just queries numpy include dirs in an incorrect fashion, should use `numpy.get_include` rocketcea: nothing special, can just switch to setuptools brian: no usage, just a string `numpy.distutils` in docs libvigraimpex: just queries numpy include dirs in an incorrect fashion, should use `numpy.get_include` gpaw: some weird monkeypatching in shell scripts of numpy's BLAS/LAPACK support, can probably just be dropped pyferret: uses Extension, can just switch to setuptools adolc: just queries numpy include dirs in an incorrect fashion, should use `numpy.get_include` theano: uses BLAS support, they should probably vendor `system_info.py` (or switch build systems) scikit-image: uses Configuration scikit-ffm: uses Configuration, but only two setup.py files - easy to consolidate in a single setup.py and switch to setuptools pywavelets: no longer uses numpy.distutils pybind11: just a code comment, no numpy.distutils usage sfepy: uses Configuration nibabel: only one usage in a test case, just drop it versioneer: just a mention in the docs cypari: just a patch for numpy < 1.11.2, can be dropped blender: just a patch that can be dropped scikit-learn: uses Configuration and build_ext modification for parallel compilation nipype: minor usage of Configuration, can be dropped guiqwt: uses Extension but no longer has Fortran code, so can switch to setuptools
The summary of this is: if Configuration is added to setuptools, 90% of packages are fine switching to setuptools. Scikit-learn is planning to follow SciPy and move to Meson. Theano is the only real consumer of numpy.distutils BLAS/LAPACK support, which they should vendor if they don't want to switch build systems. There are no other serious issues here.
I'm sure there are some more packages out there relying on Fortran support in particular, they are best off moving to Meson or CMake.
Cheers, Ralf
On Mon, Jan 17, 2022 at 3:17 PM Jerome Kieffer <Jerome.Kieffer@esrf.fr> wrote:
Hi,
I should mention many of our software projects (fabio, pyFAI, ...) rely on silx for the nested setup.py part, and not all are mentionned in this list. In the absence of alternative solution, we limit the version of setup.py for now but we are willing to help in finding a solution (our complete build stack would collapses).
Well the parallel build feature is a very much appreciated because the serial compilation of our project take dozens of minutes and our computers are HPC nodes with plenty of cores...
Thanks for pointing those projects out Jerome. It looks like you'll be fine moving to setuptools if Configuration and parallel compilation get added there. Although with those kinds of build times you may at some point want to move - Meson is significantly faster.
I find funny the way of first deprecate ... then look if any alternative solution exists.
Well, it's morer less just formalizing the status quo. `distutils` is going away, so if we do nothing then `numpy.distutils` simply also goes away. This is basically just (a) a heads up to users what is coming, and (b) giving people who need features and don't want to move build systems the time to consider pitching in with moving the things they need into setuptools.
If I understand properly, your idea is to port the nested setup.py part from numpy.distutils to setuptools ? Are people of setuptools already in the loop ?
Yes, the lead setuptools maintainer has stated that they're happy to accept any new features that make sense. Cheers, Ralf
Cheers,
Jerome
On Mon, 17 Jan 2022 10:57:40 +0100 Ralf Gommers <ralf.gommers@gmail.com> wrote:
this may need to be added to setuptools python-pot: uses private numpy.distutils compiler support; can be switched to setuptools or vendored petsc4py: small setup.py, but uses Fortran support and Configuration - should probably switch build system basemap: uses Extension, can just switch to setuptools openexr: just queries numpy include dirs in an incorrect fashion, should use `numpy.get_include` rocketcea: nothing special, can just switch to setuptools brian: no usage, just a string `numpy.distutils` in docs libvigraimpex: just queries numpy include dirs in an incorrect fashion, should use `numpy.get_include` gpaw: some weird monkeypatching in shell scripts of numpy's BLAS/LAPACK support, can probably just be dropped pyferret: uses Extension, can just switch to setuptools adolc: just queries numpy include dirs in an incorrect fashion, should use `numpy.get_include` theano: uses BLAS support, they should probably vendor `system_info.py` (or switch build systems) scikit-image: uses Configuration scikit-ffm: uses Configuration, but only two setup.py files - easy to consolidate in a single setup.py and switch to setuptools pywavelets: no longer uses numpy.distutils pybind11: just a code comment, no numpy.distutils usage sfepy: uses Configuration nibabel: only one usage in a test case, just drop it versioneer: just a mention in the docs cypari: just a patch for numpy < 1.11.2, can be dropped blender: just a patch that can be dropped scikit-learn: uses Configuration and build_ext modification for parallel compilation nipype: minor usage of Configuration, can be dropped guiqwt: uses Extension but no longer has Fortran code, so can switch to setuptools
The summary of this is: if Configuration is added to setuptools, 90% of packages are fine switching to setuptools. Scikit-learn is planning to follow SciPy and move to Meson. Theano is the only real consumer of numpy.distutils BLAS/LAPACK support, which they should vendor if they don't want to switch build systems. There are no other serious issues here.
I'm sure there are some more packages out there relying on Fortran support in particular, they are best off moving to Meson or CMake.
Cheers, Ralf
On Sun, Jan 16, 2022 at 10:40 PM Sandro Tosi <sandro.tosi@gmail.com> wrote:
4. Write a docs page with migration info on how to deal with the deprecation, and update it before the 1.23.0 release.
i think this will be extremely important. Just to give you the current POV of debian projects using numpy.distutils:
-
https://codesearch.debian.net/search?q=numpy.distutils+-package%3Anumpy&literal=0&perpkg=1
(there are 74 of them) -
https://codesearch.debian.net/search?q=from+numpy+import+.*distutils&literal=0
(only one "from numpy import distutils")
hope this could help you come up with a smooth deprecation process :)
Thanks Sandro, that is helpful. I had a look at all those 15 pages of search results:
PyPy: just a few patches for numpy <1.11.2 which can be dropped Silx: uses Configuration (i.e., nested setup.py's) Electrum: just a few patches for numpy <1.11.2 which can be dropped Setuptools: just a few patches for numpy <1.11.2 which can be dropped python-hdf4: nothing special, can be switched to setuptools pybigwig: uses `get_info('npymath')` to (optionally) build against numpy
On Sun, Jan 16, 2022 at 7:41 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
Hi all,
Now that 1.22.0 is out the door, I think it's time to deprecate `numpy.distutils` completely.
A brief summary for context: - `distutils` was deprecated in Python 3.10, and will be removed in Python 3.12 (Oct 2023 release date) - `setuptools` has re-enabled its vendored copy of `distutils` in its 60.0 release. It's incompatible with `numpy.distutils` and many more breaking changes are to be expected. - Rebasing `numpy.distutils` on top of `setuptools` will be a moving target. More importantly, `numpy.distutils` still has serious and hard to fix bugs and design issues. For example, we cannot build SciPy in parallel due to race conditions in the Fortran support code, and there's a very elusive bug where building an extension module which links in a static Fortran-only library is underlinked and breaks at runtime in dev environments.
My initial plan was to contribute Fortran support and other `numpy.distutils` features to `setuptools`, but I've personally changed my mind on that and don't plan to do that work. I think `setuptools` is moving too slow for this to be a productive exercise, and the design of distutils is fundamentally unfixable. So it's a much better investment of time to help projects migrate away from `numpy.distutils` (and hence from `setuptools`).
The tracking issue for this is https://github.com/numpy/numpy/issues/18588. Please see the comments and linked issue/thread for more details.
My proposal is: 1. deprecate `numpy.distutils` now in main. 2. keep it around for another 2 years after the 1.23.0 release. On >=3.12 it will raise an informative error when someone tries to import it. 3. Only setuptools < 60.0 is supported. This is the status as of today - unless there's an issue with Python 3.11 support, I don't see a point in spending time on keeping up with new setuptools releases, a pin is just fine. 4. Write a docs page with migration info on how to deal with the deprecation, and update it before the 1.23.0 release.
Note that the setuptools maintainers have said that they are happy to consider new features if people want to integrate them. The most popular/useful features for `numpy.distutils` are (1) Fortran support, and (2) nested setup.py support. The former is probably not a good idea to integrate, the latter may be. There aren't many packages who need BLAS/LAPACK support anymore, scikit-learn/statsmodels/etc. switched to the Cython/C API exposed by SciPy, rather than linking to BLAS/LAPACK directly.
For NumPy itself, we should move to Meson after the SciPy 1.9.0 release has gone out and ironed out all the remaining issues there. We can revisit that in 6 months or so.
Thoughts?
I've opened a PR for this: https://github.com/numpy/numpy/pull/20875 Cheers, Ralf
participants (4)
-
Jerome Kieffer -
Oscar Benjamin -
Ralf Gommers -
Sandro Tosi