reducing effort spent on wheel builds?
Hey all, This whole thread is quite interesting: https://twitter.com/zooba/status/1415440484181417998. Given how much effort we are spending on really niche wheel builds, I’m wondering if we should just draw a line somewhere: - we do what we do now for the main platforms: Windows, Linux (x86, aarch64), macOS, *but*: - no wheels for ppc64le - no wheels for Alpine Linux - no wheels for PyPy - no wheels for Raspberry Pi, AIX or whatever other niche thing comes next. - drop 32-bit Linux in case it is becoming an annoyance. This is not an actual proposal (yet) and I should sleep on this some more, but I've seen Chuck and Matti burn a lot of time on the numpy-wheels repo again recently, and I've done the same for SciPy. The situation is not very sustainable and needs a rethink. The current recipe is "someone who cares about a platform writes a PEP, then pip/wheel add a platform tag for it (very little work), and then the maintainers of each Python package are now responsible for wheel builds (a ton of work)". Most of these platforms have package managers, which are all more capable than pip et al., and if they don't then wheels can be hosted elsewhere (example: https://www.piwheels.org/). And then there's Conda, Nix, Spack, etc. too of course. Drawing a line somewhere distributes the workload, where packagers who care about some platform and have better tools at hand can do the packaging, and maintainers can go do something with more impact like write new code or review PRs. <end of brainwave> Cheers, Ralf
FWIW, here's a big fat +1 from me for spreading the load. I'd even advocate for trimming the CI and going for "We officially support this (small) list of platforms and gladly accept patches for anything else as long as they do not break the officially supported ones". ISTM the list of supported platforms (both wheels and CI) has grown too large and seems to only grow in time. The user requests are perfectly understandable, everyone wants to be in the press-the-button-and-it-works world, but the maintainer/RM effort seems to be crossing over to being too much. A perfect picture in this regard is probably what we had a couple of years ago with Gohlke Windows wheels. Christoph was doing a great job maintaining the wheels and sending patches. For more exotic platforms, there simply has to be a champion. Or if some entity wants to fund the work for some platform, great --- this should also live somewhere outside of the main repo/CI/pypi. Whether to upload sdists to PYPI --- this is a bit orthogonal, but indeed maybe it's best to only upload the small set of wheels indeed. sdists are just stored in the GH releases and it's not a big deal to grab them from GH (or even pip install from the release tag directly). This would probably improve user experience too (no more cryptic errors from attempting to compile from source on an unsuspecting user). My 2p, Evgeni On Thu, Jul 15, 2021 at 1:22 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
Hey all,
This whole thread is quite interesting: https://twitter.com/zooba/status/1415440484181417998. Given how much effort we are spending on really niche wheel builds, I’m wondering if we should just draw a line somewhere:
we do what we do now for the main platforms: Windows, Linux (x86, aarch64), macOS, *but*: no wheels for ppc64le no wheels for Alpine Linux no wheels for PyPy no wheels for Raspberry Pi, AIX or whatever other niche thing comes next. drop 32-bit Linux in case it is becoming an annoyance.
This is not an actual proposal (yet) and I should sleep on this some more, but I've seen Chuck and Matti burn a lot of time on the numpy-wheels repo again recently, and I've done the same for SciPy. The situation is not very sustainable and needs a rethink.
The current recipe is "someone who cares about a platform writes a PEP, then pip/wheel add a platform tag for it (very little work), and then the maintainers of each Python package are now responsible for wheel builds (a ton of work)". Most of these platforms have package managers, which are all more capable than pip et al., and if they don't then wheels can be hosted elsewhere (example: https://www.piwheels.org/). And then there's Conda, Nix, Spack, etc. too of course.
Drawing a line somewhere distributes the workload, where packagers who care about some platform and have better tools at hand can do the packaging, and maintainers can go do something with more impact like write new code or review PRs.
<end of brainwave>
Cheers, Ralf
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
I'd be +1 on reducing the number of wheels to reduce maintainer effort (esp. the 32 bit builds). However, I think reducing the CI dedication (+maintainer effort towards that) to those minority platforms would inevitably increase the risk of numpy/scipy failing to work on them, and possibly risk not being installable on them. Would we want to drop them without having a positive assurance that the minority platform/package maintainers would keep checking for numpy/scipy health; I guess one can make that choice when you know just how many times those exotic wheels are downloaded (modulo CI runs). I'd like to see sdists being kept on PyPI. Those minority platforms may keep patching scipy/numpy source to keep it installable from source, and pip knows where to find it. I suspect the majority platforms will have all the wheels they need, so only the small set of exotic platforms will use sdist, with those users being more capable of dealing with the aftermath. On Thu, 15 Jul 2021 at 20:53, Evgeni Burovski <evgeny.burovskiy@gmail.com> wrote:
FWIW, here's a big fat +1 from me for spreading the load. I'd even advocate for trimming the CI and going for "We officially support this (small) list of platforms and gladly accept patches for anything else as long as they do not break the officially supported ones". ISTM the list of supported platforms (both wheels and CI) has grown too large and seems to only grow in time.
The user requests are perfectly understandable, everyone wants to be in the press-the-button-and-it-works world, but the maintainer/RM effort seems to be crossing over to being too much.
A perfect picture in this regard is probably what we had a couple of years ago with Gohlke Windows wheels. Christoph was doing a great job maintaining the wheels and sending patches. For more exotic platforms, there simply has to be a champion. Or if some entity wants to fund the work for some platform, great --- this should also live somewhere outside of the main repo/CI/pypi.
Whether to upload sdists to PYPI --- this is a bit orthogonal, but indeed maybe it's best to only upload the small set of wheels indeed. sdists are just stored in the GH releases and it's not a big deal to grab them from GH (or even pip install from the release tag directly). This would probably improve user experience too (no more cryptic errors from attempting to compile from source on an unsuspecting user).
My 2p,
Evgeni
On Thu, Jul 15, 2021 at 1:22 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
Hey all,
This whole thread is quite interesting:
https://twitter.com/zooba/status/1415440484181417998. Given how much effort we are spending on really niche wheel builds, I’m wondering if we should just draw a line somewhere:
we do what we do now for the main platforms: Windows, Linux (x86,
no wheels for ppc64le no wheels for Alpine Linux no wheels for PyPy no wheels for Raspberry Pi, AIX or whatever other niche thing comes next. drop 32-bit Linux in case it is becoming an annoyance.
This is not an actual proposal (yet) and I should sleep on this some more, but I've seen Chuck and Matti burn a lot of time on the numpy-wheels repo again recently, and I've done the same for SciPy. The situation is not very sustainable and needs a rethink.
The current recipe is "someone who cares about a platform writes a PEP,
aarch64), macOS, *but*: then pip/wheel add a platform tag for it (very little work), and then the maintainers of each Python package are now responsible for wheel builds (a ton of work)". Most of these platforms have package managers, which are all more capable than pip et al., and if they don't then wheels can be hosted elsewhere (example: https://www.piwheels.org/). And then there's Conda, Nix, Spack, etc. too of course.
Drawing a line somewhere distributes the workload, where packagers who
care about some platform and have better tools at hand can do the packaging, and maintainers can go do something with more impact like write new code or review PRs.
<end of brainwave>
Cheers, Ralf
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
-- _____________________________________ Dr. Andrew Nelson _____________________________________
However, I think reducing the CI dedication (+maintainer effort towards that) to those minority platforms would inevitably increase the risk of numpy/scipy failing to work on them, and possibly risk not being installable on them. Would we want to drop them without having a positive assurance that the minority platform/package maintainers would keep checking for numpy/scipy health
Nothing blocks a platform champion from running the CI on their scipy fork. This way, the burden is on them, not on individual contributors (including first-time contributors who only deal with common platforms). чт, 15 июл. 2021 г., 15:22 Andrew Nelson <andyfaff@gmail.com>:
I'd be +1 on reducing the number of wheels to reduce maintainer effort (esp. the 32 bit builds). However, I think reducing the CI dedication (+maintainer effort towards that) to those minority platforms would inevitably increase the risk of numpy/scipy failing to work on them, and possibly risk not being installable on them. Would we want to drop them without having a positive assurance that the minority platform/package maintainers would keep checking for numpy/scipy health; I guess one can make that choice when you know just how many times those exotic wheels are downloaded (modulo CI runs).
I'd like to see sdists being kept on PyPI. Those minority platforms may keep patching scipy/numpy source to keep it installable from source, and pip knows where to find it. I suspect the majority platforms will have all the wheels they need, so only the small set of exotic platforms will use sdist, with those users being more capable of dealing with the aftermath.
On Thu, 15 Jul 2021 at 20:53, Evgeni Burovski <evgeny.burovskiy@gmail.com> wrote:
FWIW, here's a big fat +1 from me for spreading the load. I'd even advocate for trimming the CI and going for "We officially support this (small) list of platforms and gladly accept patches for anything else as long as they do not break the officially supported ones". ISTM the list of supported platforms (both wheels and CI) has grown too large and seems to only grow in time.
The user requests are perfectly understandable, everyone wants to be in the press-the-button-and-it-works world, but the maintainer/RM effort seems to be crossing over to being too much.
A perfect picture in this regard is probably what we had a couple of years ago with Gohlke Windows wheels. Christoph was doing a great job maintaining the wheels and sending patches. For more exotic platforms, there simply has to be a champion. Or if some entity wants to fund the work for some platform, great --- this should also live somewhere outside of the main repo/CI/pypi.
Whether to upload sdists to PYPI --- this is a bit orthogonal, but indeed maybe it's best to only upload the small set of wheels indeed. sdists are just stored in the GH releases and it's not a big deal to grab them from GH (or even pip install from the release tag directly). This would probably improve user experience too (no more cryptic errors from attempting to compile from source on an unsuspecting user).
My 2p,
Evgeni
On Thu, Jul 15, 2021 at 1:22 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
Hey all,
This whole thread is quite interesting:
https://twitter.com/zooba/status/1415440484181417998. Given how much effort we are spending on really niche wheel builds, I’m wondering if we should just draw a line somewhere:
we do what we do now for the main platforms: Windows, Linux (x86,
no wheels for ppc64le no wheels for Alpine Linux no wheels for PyPy no wheels for Raspberry Pi, AIX or whatever other niche thing comes next. drop 32-bit Linux in case it is becoming an annoyance.
This is not an actual proposal (yet) and I should sleep on this some more, but I've seen Chuck and Matti burn a lot of time on the numpy-wheels repo again recently, and I've done the same for SciPy. The situation is not very sustainable and needs a rethink.
The current recipe is "someone who cares about a platform writes a PEP,
aarch64), macOS, *but*: then pip/wheel add a platform tag for it (very little work), and then the maintainers of each Python package are now responsible for wheel builds (a ton of work)". Most of these platforms have package managers, which are all more capable than pip et al., and if they don't then wheels can be hosted elsewhere (example: https://www.piwheels.org/). And then there's Conda, Nix, Spack, etc. too of course.
Drawing a line somewhere distributes the workload, where packagers who
care about some platform and have better tools at hand can do the packaging, and maintainers can go do something with more impact like write new code or review PRs.
<end of brainwave>
Cheers, Ralf
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
-- _____________________________________ Dr. Andrew Nelson
_____________________________________ _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
On 15/7/21 1:21 pm, Ralf Gommers wrote:
Hey all,
I've seen Chuck and Matti burn a lot of time on the numpy-wheels repo again recently, and I've done the same for SciPy. ...
Cheers, Ralf
Since my name was mentioned, the things I have spent time on for wheel packaging in order of time spent (not scientifically measured rather looking back at the closed PRs and thinking "oh yeah, that was painful") have been - Updating manylnux to move away from 10 year old glibc on linux (still stuck and not clear how to finish it [0]) - Moving from travis-ci.org to travis-ci.com (with the panic around build credits) and from Appveyor to github actions/azure pipelines - Moving from rackspace's wheel hosting to anaconda.org - Working around CI failures with aarch64 for linux, mainly due to shortcomings in the free CI providers - Bugs in dependencies: avoiding buggy Accelerate when building NumPy and debugging Windows/aarch64 problems with OpenBLAS - Updating OpenBLAS versions - Shepherding Apple M1 hardware through the manylinux/multibuild/wheel pipeline (all the hard work was done by others) - Trying to make 64-bit interfaces for OpenBLAS work (99.9% done by Chuck) - Updating PyPy versions Only the last one, which was actually the least painful, would be helped by Ralf's list. On the other hand, packaging is made harder as more technologies go into a wheel build. The twitter thread started with "SciPy added a required dependency on a technology which broke things, but people stepped up to fix the problem quickly" and morphed into "lets drop wheels for lesser used platforms". I am not sure the discussion should have moved away from the first point so quickly. Perhaps there should be some discussion of the cost of adding new build dependencies and somehow making the dependency conditional for a release or two until all the worst kinks are worked out. For the record, I am +1 on removing sdists from PyPI until pip changes its default to --only-binary :all: [1] Matti [0] https://github.com/pypa/manylinux/issues/1012 [1] https://github.com/pypa/pip/issues/9140
On Thu, Jul 15, 2021 at 2:41 PM Matti Picus <matti.picus@gmail.com> wrote:
On 15/7/21 1:21 pm, Ralf Gommers wrote:
Hey all,
I've seen Chuck and Matti burn a lot of time on the numpy-wheels repo again recently, and I've done the same for SciPy. ...
Cheers, Ralf
Since my name was mentioned, the things I have spent time on for wheel packaging in order of time spent (not scientifically measured rather looking back at the closed PRs and thinking "oh yeah, that was painful") have been
- Updating manylnux to move away from 10 year old glibc on linux (still stuck and not clear how to finish it [0])
- Moving from travis-ci.org to travis-ci.com (with the panic around build credits) and from Appveyor to github actions/azure pipelines
- Moving from rackspace's wheel hosting to anaconda.org
- Working around CI failures with aarch64 for linux, mainly due to shortcomings in the free CI providers
- Bugs in dependencies: avoiding buggy Accelerate when building NumPy and debugging Windows/aarch64 problems with OpenBLAS
- Updating OpenBLAS versions
- Shepherding Apple M1 hardware through the manylinux/multibuild/wheel pipeline (all the hard work was done by others)
- Trying to make 64-bit interfaces for OpenBLAS work (99.9% done by Chuck)
- Updating PyPy versions
Thanks Matti! This list is super helpful.
Only the last one, which was actually the least painful, would be helped by Ralf's list.
Not so sure about that - probably the single biggest pain points are CI providers (especially the exotic ones) and OpenBLAS - a dependency we struggle with mostly because of wheels. Without ppc64le and s390x (forgot that one) we wouldn't need TravisCI at all for example, and we would have less work on https://github.com/MacPython/openblas-libs too. On the other hand, packaging is made harder as more technologies go into
a wheel build. The twitter thread started with "SciPy added a required dependency on a technology which broke things, but people stepped up to fix the problem quickly" and morphed into "lets drop wheels for lesser used platforms". I am not sure the discussion should have moved away from the first point so quickly. Perhaps there should be some discussion of the cost of adding new build dependencies and somehow making the dependency conditional for a release or two until all the worst kinks are worked out.
Not quite the right mailing list, but: it *is* an optional dependency in scipy 1.7.0, and there has been no proposal so far to make it required.
For the record, I am +1 on removing sdists from PyPI until pip changes its default to --only-binary :all: [1]
Matti
Ah yes:( That just seems like a major oversight in the perennial manylinux spec. If no one fixes it, guess we wait till we can jump to `_2_??` with a higher GCC requirement. Cheers, Ralf
On Thu, Jul 15, 2021 at 3:45 PM Kevin Sheppard <kevin.k.sheppard@gmail.com> wrote:
When thinking about supporting a platform, it seems reasonable to consider other sources for pre-compiled binaries, e.g., conda and especially conda-forge. Conda-forge has
linux-ppc64le v1.7.0
linux-64 v1.7.0
linux-aarch64 v1.7.0
osx-arm64 v1.7.0
osx-64 v1.7.0
win-32 v1.2.1
win-64 v1.7.0
If this list included PyPy that it would be virtually complete.
Conda-forge supports PyPy, and the NumPy build seems to have a problem but that's being worked on at the moment: https://github.com/conda-forge/numpy-feedstock/pull/238 Cheers, Ralf
It seems reasonable to direct users to conda-forge if they are unable to build a package for a less common platform.
Building on conda-forge is often easier than building a wheel for PyPI due to the substantial infrastructure that the conda-forge team provides to automatically wire up CI for a lot of platforms.
Kevin
On 15/7/21 4:11 pm, Ralf Gommers wrote:
On Thu, Jul 15, 2021 at 2:41 PM Matti Picus <matti.picus@gmail.com <mailto:matti.picus@gmail.com>> wrote:
On the other hand, packaging is made harder as more technologies go into a wheel build. ... Perhaps there should be some discussion of the cost of adding new build dependencies and somehow making the dependency conditional ...
Not quite the right mailing list, but: it *is* an optional dependency in scipy 1.7.0, and there has been no proposal so far to make it required.
Cheers, Ralf
Thanks for the clarification, sorry for the misunderstanding Matti
Just a note on:
For the record, I am +1 on removing sdists from PyPI until pip changes its default to --only-binary :all: [1]
I agree that the defaults for pip are unfortunate (and indeed the legacy of pip doing, well, a lot, (i.e. building and installing and package managing and dependencies, and ...) with one interface. However, There's a long tradition of sdists on PyPi -- and PyPi is used, for the most part, as the source of sdists for other systems (conda-forge for example). I did just check, and numpy is an exception -- it's pointing to gitHub: source: url: https://github.com/numpy/numpy/releases/download/v{{ version }}/numpy-{{ version }}.tar.gz But others may be counting on sdists on PyPi. Also, an sdist is not always the same as a gitHub release -- there is some "magic" in building it -- it's not just a copy of the repo. Again, numpy may be building its releases as an sdist (or it just doesn't. matter), but something to keep in mind. Another thought is to only support platforms that have a committed maintainer -- I think that's how Python itself does it. The more obscure platforms are only supported if someone steps up to support them (I suppose that's technically true for all platforms, but not hard to find someone on the existing core dev team to support the majors). This can be a bit tricky, as the users of a platform may not have the skills to maintain the builds, but it seems fair enough to only support platforms that someone cares enough about to do the work. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 16/7/21 9:11 pm, Chris Barker wrote:
Just a note on:
For the record, I am +1 on removing sdists from PyPI until pip changes its default to --only-binary :all: [1]
I agree that the defaults for pip are unfortunate (and indeed the legacy of pip doing, well, a lot, (i.e. building and installing and package managing and dependencies, and ...) with one interface.
However, There's a long tradition of sdists on PyPi -- and PyPi is used, for the most part, as the source of sdists for other systems (conda-forge for example). I did just check, and numpy is an exception -- it's pointing to gitHub:
source: url: https://github.com/numpy/numpy/releases/download/v{{ <https://github.com/numpy/numpy/releases/download/v{{> version }}/numpy-{{ version }}.tar.gz
But others may be counting on sdists on PyPi.
Also, an sdist is not always the same as a gitHub release -- there is some "magic" in building it -- it's not just a copy of the repo. Again, numpy may be building its releases as an sdist (or it just doesn't. matter), but something to keep in mind.
Another thought is to only support platforms that have a committed maintainer -- I think that's how Python itself does it. The more obscure platforms are only supported if someone steps up to support them (I suppose that's technically true for all platforms, but not hard to find someone on the existing core dev team to support the majors). This can be a bit tricky, as the users of a platform may not have the skills to maintain the builds, but it seems fair enough to only support platforms that someone cares enough about to do the work.
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov <mailto:Chris.Barker@noaa.gov>
Just an empty response since this ended up in my spam filter, and I am probably not the only one. Matti
I am pretty -1 on removing sdists. At least for Matplotlib we have a number of users on AIX who rely on the the sdists (I know they exist because when our build breaks they send us patches to un-break it) for their installation. I strongly suspect that numpy also has a fair number of users who are willing and able to compile from source on niche systems and it does seem like a good idea to me to break them. I like Ralf's proposal, but would amend it with "unless someone stands up and puts their name on ensuring the wheels exist for platform <X>" with the understanding that if they have to step away support for that platform stops until someone else is willing to put their name on it. This may be a good candidate for the new SPEC process to handle? Just like supported versions of Python, we should have consistent platform support (and processes for how we decide on / provide that support) across the community (thinking with my Matplotlib and h5py hats here). Tom On Mon, Aug 9, 2021 at 9:51 AM Matti Picus <matti.picus@gmail.com> wrote:
On 16/7/21 9:11 pm, Chris Barker wrote:
Just a note on:
For the record, I am +1 on removing sdists from PyPI until pip changes its default to --only-binary :all: [1]
I agree that the defaults for pip are unfortunate (and indeed the legacy of pip doing, well, a lot, (i.e. building and installing and package managing and dependencies, and ...) with one interface.
However, There's a long tradition of sdists on PyPi -- and PyPi is used, for the most part, as the source of sdists for other systems (conda-forge for example). I did just check, and numpy is an exception -- it's pointing to gitHub:
source: url: https://github.com/numpy/numpy/releases/download/v{{ <https://github.com/numpy/numpy/releases/download/v{{> version }}/numpy-{{ version }}.tar.gz
But others may be counting on sdists on PyPi.
Also, an sdist is not always the same as a gitHub release -- there is some "magic" in building it -- it's not just a copy of the repo. Again, numpy may be building its releases as an sdist (or it just doesn't. matter), but something to keep in mind.
Another thought is to only support platforms that have a committed maintainer -- I think that's how Python itself does it. The more obscure platforms are only supported if someone steps up to support them (I suppose that's technically true for all platforms, but not hard to find someone on the existing core dev team to support the majors). This can be a bit tricky, as the users of a platform may not have the skills to maintain the builds, but it seems fair enough to only support platforms that someone cares enough about to do the work.
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov <mailto:Chris.Barker@noaa.gov>
Just an empty response since this ended up in my spam filter, and I am probably not the only one.
Matti
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
-- Thomas Caswell tcaswell@gmail.com
I spent so much time updating the wheels builds to 64 bit BLAS mostly because - I needed to actually understand how multibuild worked (and docs are minimal). - I don't know powershell (and docs are hard to find). - I don't know azure works at a lower level (and docs are hard to find). And I thought it would all be drop in easy :) What that indicates to me is that we could use a build expert and simpler infrastructure. Anaconda no doubt has people far more knowledgeable about such matters than I am. As to the twitter thread, what is the proposal(s)? I'm perfectly happy to leave support for more exotic platforms to people who have the hardware and make use of it, but no doubt such people have the opposite problem of not knowing much about NumPy. Chuck On Thu, Jul 15, 2021 at 4:22 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
Hey all,
This whole thread is quite interesting: https://twitter.com/zooba/status/1415440484181417998. Given how much effort we are spending on really niche wheel builds, I’m wondering if we should just draw a line somewhere:
- we do what we do now for the main platforms: Windows, Linux (x86, aarch64), macOS, *but*: - no wheels for ppc64le - no wheels for Alpine Linux - no wheels for PyPy - no wheels for Raspberry Pi, AIX or whatever other niche thing comes next. - drop 32-bit Linux in case it is becoming an annoyance.
This is not an actual proposal (yet) and I should sleep on this some more, but I've seen Chuck and Matti burn a lot of time on the numpy-wheels repo again recently, and I've done the same for SciPy. The situation is not very sustainable and needs a rethink.
The current recipe is "someone who cares about a platform writes a PEP, then pip/wheel add a platform tag for it (very little work), and then the maintainers of each Python package are now responsible for wheel builds (a ton of work)". Most of these platforms have package managers, which are all more capable than pip et al., and if they don't then wheels can be hosted elsewhere (example: https://www.piwheels.org/). And then there's Conda, Nix, Spack, etc. too of course.
Drawing a line somewhere distributes the workload, where packagers who care about some platform and have better tools at hand can do the packaging, and maintainers can go do something with more impact like write new code or review PRs.
<end of brainwave>
Cheers, Ralf
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
I spent so much time updating the wheels builds to 64 bit BLAS mostly because
- I needed to actually understand how multibuild worked (and docs are minimal). - I don't know powershell (and docs are hard to find). - I don't know azure works at a lower level (and docs are hard to find).
And I thought it would all be drop in easy :) What that indicates to me is that we could use a build expert and simpler infrastructure. Anaconda no doubt has people far more knowledgeable about such matters than I am.
As to the twitter thread, what is the proposal(s)? I'm perfectly happy to leave support for more exotic platforms to people who have the hardware and make use of it, but no doubt such people have the opposite problem of not knowing much about NumPy.
Chuck
On Thu, Jul 15, 2021 at 4:22 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
Hey all,
This whole thread is quite interesting: https://twitter.com/zooba/status/1415440484181417998. Given how much effort we are spending on really niche wheel builds, I’m wondering if we should just draw a line somewhere:
- we do what we do now for the main platforms: Windows, Linux (x86, aarch64), macOS, *but*: - no wheels for ppc64le - no wheels for Alpine Linux - no wheels for PyPy - no wheels for Raspberry Pi, AIX or whatever other niche thing comes next. - drop 32-bit Linux in case it is becoming an annoyance.
This is not an actual proposal (yet) and I should sleep on this some more, but I've seen Chuck and Matti burn a lot of time on the numpy-wheels repo again recently, and I've done the same for SciPy. The situation is not very sustainable and needs a rethink.
The current recipe is "someone who cares about a platform writes a PEP, then pip/wheel add a platform tag for it (very little work), and then the maintainers of each Python package are now responsible for wheel builds (a ton of work)". Most of these platforms have package managers, which are all more capable than pip et al., and if they don't then wheels can be hosted elsewhere (example: https://www.piwheels.org/). And then there's Conda, Nix, Spack, etc. too of course.
Drawing a line somewhere distributes the workload, where packagers who care about some platform and have better tools at hand can do the packaging, and maintainers can go do something with more impact like write new code or review PRs.
<end of brainwave>
Cheers, Ralf
Let me add that distutils brings it's own set of problems. A better build system could help make building wheels simpler on various platforms. But
On Thu, Jul 15, 2021 at 8:02 AM Charles R Harris <charlesr.harris@gmail.com> wrote: then, someone would also need to become expert in that build system. Chuck
On Thu, Jul 15, 2021 at 4:15 PM Charles R Harris <charlesr.harris@gmail.com> wrote:
On Thu, Jul 15, 2021 at 8:02 AM Charles R Harris < charlesr.harris@gmail.com> wrote:
I spent so much time updating the wheels builds to 64 bit BLAS mostly because
- I needed to actually understand how multibuild worked (and docs are minimal). - I don't know powershell (and docs are hard to find). - I don't know azure works at a lower level (and docs are hard to find).
Yes, this is a problem. powershell/bash + no docs is not a recipe for success. Longer term we could move it over to cibuildwheel which has decent docs, but that in itself is a lot of work so it won't happen overnight.
And I thought it would all be drop in easy :) What that indicates to me is that we could use a build expert and simpler infrastructure. Anaconda no doubt has people far more knowledgeable about such matters than I am.
As to the twitter thread, what is the proposal(s)?
There's no proposal yet, still stewing a bit on it. Although maybe the first part of one is ready: no more new wheel flavors for things with <<0.5% of the user base. I'm perfectly happy to leave support for more exotic platforms to people
who have the hardware and make use of it, but no doubt such people have the opposite problem of not knowing much about NumPy.
Let me add that distutils brings it's own set of problems. A better build system could help make building wheels simpler on various platforms. But then, someone would also need to become expert in that build system.
Working on that one:) Cheers, Ralf
participants (8)
-
Andrew Nelson -
Charles R Harris -
Chris Barker -
Evgeni Burovski -
Kevin Sheppard -
Matti Picus -
Ralf Gommers -
Thomas Caswell