[Numpy-discussion] reducing effort spent on wheel builds?

[Matti Picus]

On 15/7/21 1:21 pm, Ralf Gommers wrote:

> Hey all,
>
> I've seen Chuck and Matti burn a lot of time on the numpy-wheels repo 
> again recently, and I've done the same for SciPy. ...
>
> Cheers,
> Ralf


Since my name was mentioned, the things I have spent time on for wheel 
packaging in order of time spent (not scientifically measured rather 
looking back at the closed PRs and thinking "oh yeah, that was painful") 
have been

- Updating manylnux to move away from 10 year old glibc on linux (still 
stuck and not clear how to finish it [0])

- Moving from travis-ci.org to travis-ci.com (with the panic around 
build credits) and from Appveyor to github actions/azure pipelines

- Moving from rackspace's wheel hosting to anaconda.org

- Working around CI failures with aarch64 for linux, mainly due to 
shortcomings in the free CI providers

- Bugs in dependencies: avoiding buggy Accelerate when building NumPy 
and debugging Windows/aarch64 problems with OpenBLAS

- Updating OpenBLAS versions

- Shepherding Apple M1 hardware through the manylinux/multibuild/wheel 
pipeline (all the hard work was done by others)

- Trying to make 64-bit interfaces for OpenBLAS work (99.9% done by Chuck)

- Updating PyPy versions


Only the last one, which was actually the least painful, would be helped 
by Ralf's list.


On the other hand, packaging is made harder as more technologies go into 
a wheel build. The twitter thread started with "SciPy added a required 
dependency on a technology which broke things, but people stepped up to 
fix the problem quickly" and morphed into "lets drop wheels for lesser 
used platforms". I am not sure the discussion should have moved away 
from the first point so quickly. Perhaps there should be some discussion 
of the cost of adding new build dependencies and somehow making the 
dependency conditional for a release or two until all the worst kinks 
are worked out.


For the record, I am +1 on removing sdists from PyPI until pip changes 
its default to --only-binary :all: [1]

Matti


[0] https://github.com/pypa/manylinux/issues/1012

[1] https://github.com/pypa/pip/issues/9140