Hi, I am writing on behalf of my team from Endava company. Our task is to work with opensource community, and setup multiple applications for use on win11 arm64 machines. One of the tasks is to understand the problems and if possible help NumPy team in setting up win11 arm64 compatible wheel on the PyPi. I saw that there were some earlier discussions about this subject, and as I understood problem was that at that moment this configuration was not so common and there were no appropriate CI slaves to create the wheel. Today there are more devices in this configuration available on the market and additional are announced, so I wanted to check if there are plans for creating this wheel, and if we can somehow help with that work? Regards, Slobodan
On Thu, Jul 11, 2024, 22:08 slobodan.miletic--- via NumPy-Discussion < numpy-discussion@python.org> wrote:
Hi,
I am writing on behalf of my team from Endava company. Our task is to work with opensource community, and setup multiple applications for use on win11 arm64 machines. One of the tasks is to understand the problems and if possible help NumPy team in setting up win11 arm64 compatible wheel on the PyPi. I saw that there were some earlier discussions about this subject, and as I understood problem was that at that moment this configuration was not so common and there were no appropriate CI slaves to create the wheel. Today there are more devices in this configuration available on the market and additional are announced, so I wanted to check if there are plans for creating this wheel, and if we can somehow help with that work?
This is discussed in detail in https://github.com/numpy/numpy/issues/22530. TLDR - we need CI providers to offer win arm before it's sensible to make wheels. Firstly because we need to make them (less secure to upload wheels not made on CI infra), secondly because we need to test them. If less tested wheels give rise to lots of issues reported by the community, then this generates large maintainer workload.
Hi, On Thu, Jul 11, 2024 at 1:16 PM Andrew Nelson <andyfaff@gmail.com> wrote:
On Thu, Jul 11, 2024, 22:08 slobodan.miletic--- via NumPy-Discussion <numpy-discussion@python.org> wrote:
Hi,
I am writing on behalf of my team from Endava company. Our task is to work with opensource community, and setup multiple applications for use on win11 arm64 machines. One of the tasks is to understand the problems and if possible help NumPy team in setting up win11 arm64 compatible wheel on the PyPi. I saw that there were some earlier discussions about this subject, and as I understood problem was that at that moment this configuration was not so common and there were no appropriate CI slaves to create the wheel. Today there are more devices in this configuration available on the market and additional are announced, so I wanted to check if there are plans for creating this wheel, and if we can somehow help with that work?
This is discussed in detail in https://github.com/numpy/numpy/issues/22530. TLDR - we need CI providers to offer win arm before it's sensible to make wheels. Firstly because we need to make them (less secure to upload wheels not made on CI infra), secondly because we need to test them. If less tested wheels give rise to lots of issues reported by the community, then this generates large maintainer workload.
Would it make sense for a company to set up and maintain some self-hosted workflow runners? https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-ho... Cheers, Matthew
On Thu, Jul 11, 2024 at 2:20 PM Andrew Nelson <andyfaff@gmail.com> wrote:
On Thu, Jul 11, 2024, 22:08 slobodan.miletic--- via NumPy-Discussion < numpy-discussion@python.org> wrote:
Hi,
I am writing on behalf of my team from Endava company. Our task is to work with opensource community, and setup multiple applications for use on win11 arm64 machines. One of the tasks is to understand the problems and if possible help NumPy team in setting up win11 arm64 compatible wheel on the PyPi. I saw that there were some earlier discussions about this subject, and as I understood problem was that at that moment this configuration was not so common and there were no appropriate CI slaves to create the wheel. Today there are more devices in this configuration available on the market and additional are announced, so I wanted to check if there are plans for creating this wheel, and if we can somehow help with that work?
Thanks for offering help Slobodan! Contributions here are welcome - and it's hopefully not too difficult if you have a dev setup for Windows on Arm locally.
This is discussed in detail in https://github.com/numpy/numpy/issues/22530. TLDR - we need CI providers to offer win arm before it's sensible to make wheels. Firstly because we need to make them (less secure to upload wheels not made on CI infra), secondly because we need to test them. If less tested wheels give rise to lots of issues reported by the community, then this generates large maintainer workload.
A good start to get this rolling could be: 1. Address open issues for Windows on Arm. There's only one right now ( https://github.com/numpy/numpy/issues/25626) - we'd be happy to accept a PR for that, we don't need CI to validate it. 2. Setting up a single CI job that builds an arm64 wheel on x86-64 (build only, no test - unless emulation works). gh-22530 has comments saying it should work. Once those things are done, we can consider uploading wheels to https://anaconda.org/scientific-python-nightly-wheels/numpy/. Cheers, Ralf
As the bug fix is merged we are starting the investigation on the CI job. I have a few questions about this: 1) Are there some additional instructions for making and running the numpy CI jobs and cross compilation available in the documentation? 2) Do we need to have arm64 scipy-openblas released to build numpy wheel?
On 12/08/2024 11:55, slobodan.miletic--- via NumPy-Discussion wrote:
As the bug fix is merged we are starting the investigation on the CI job. I have a few questions about this: 1) Are there some additional instructions for making and running the numpy CI jobs and cross compilation available in the documentation? 2) Do we need to have arm64 scipy-openblas released to build numpy wheel? _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: matti.picus@gmail.com
1. You can look at .github/workflows/linux_qemu.yml that use qemu/docker to run. AFAIK there is no documentation about cross-compilation. 2. It would be preferable to build a scipy-openblas wheel but we could use a "vanilla" build of OpenBLAS as well. We do expect to use 64-bit interfaces, but scipy does not do so yet, so we would need two builds. I don't know how @gholke compiles OpenBLAS/NumPy and where there is a gfortran for windows arm64, maybe worthwhile asking? Matti
On Mon, Aug 12, 2024 at 12:23 PM Matti Picus <matti.picus@gmail.com> wrote:
On 12/08/2024 11:55, slobodan.miletic--- via NumPy-Discussion wrote:
As the bug fix is merged we are starting the investigation on the CI job. I have a few questions about this: 1) Are there some additional instructions for making and running the numpy CI jobs and cross compilation available in the documentation? 2) Do we need to have arm64 scipy-openblas released to build numpy wheel? _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: matti.picus@gmail.com
1. You can look at .github/workflows/linux_qemu.yml that use qemu/docker to run. AFAIK there is no documentation about cross-compilation.
There is: https://numpy.org/devdocs/building/cross_compilation.html https://cibuildwheel.pypa.io/en/stable/faq/#windows-arm64 says that cibuildwheel only supports `setuptools` and `setuptools-rust` so far, so that's not particularly helpful yet. Neither is the `linux_qemu.yml` example I think, that's the furthest away from what we need I think (unless I missed something and QEMU or Docker actually supports Windows on Arm). I'm not sure if this should go the `crossenv` way (see https://github.com/benfogle/crossenv), or a regular cross compile with a cross file and build/host envs. I think the latter won't quite work until PEP 739 (https://peps.python.org/pep-0739/) lands. 2. It would be preferable to build a scipy-openblas wheel but we could
use a "vanilla" build of OpenBLAS as well. We do expect to use 64-bit interfaces, but scipy does not do so yet, so we would need two builds. I don't know how @gholke compiles OpenBLAS/NumPy and where there is a gfortran for windows arm64, maybe worthwhile asking?
I recommend getting step (1) to work first, without any BLAS library. That may be challenging enough - if there's a blocker, then there is no point looking into cross-compiling OpenBLAS before native arm64 CI runners become available. Cheers, Ralf
Matti
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ralf.gommers@gmail.com
participants (5)
-
Andrew Nelson -
Matthew Brett -
Matti Picus -
Ralf Gommers -
slobodan.miletic＠endava.com