Numpy binary wheels and CI for win/arm64 platform
Hello, Numpy v1.21.2 <https://github.com/numpy/numpy/releases/tag/v1.21.2> added support for windows/arm64 platforms but we still don't have any systems in place to produce binary wheels or test win/arm64 packages. I think it will be good to start looking into this. CPython has an official buildbot worker running for win/arm64 and official python support for the platform will be available from the 3.11 release. It is not yet clear to me how the build and CI system for numpy is deployed and how to enable support for a new platform like win/arm64. One of the main issues in supporting win/arm64 build would be due to the lack of win/arm64 VMs available on the cloud. But I see we have been producing binary wheels for Apple M1 platforms on pypi and conda repository for some time which also lacks the cloud VM support. I think we could take some learnings from Apple M1 support and look at how a similar strategy can be used for win/arm64. I would like to hear if anyone has any thoughts on this topic. Also, any pointers to understand numpy wheel generation and CI flow for similar platforms would be helpful as well. Thanks, Niyas
On Tue, Nov 2, 2021 at 1:54 PM Niyas Sait <niyas.sait@linaro.org> wrote:
Hello,
Numpy v1.21.2 <https://github.com/numpy/numpy/releases/tag/v1.21.2> added support for windows/arm64 platforms but we still don't have any systems in place to produce binary wheels or test win/arm64 packages. I think it will be good to start looking into this. CPython has an official buildbot worker running for win/arm64 and official python support for the platform will be available from the 3.11 release.
It is not yet clear to me how the build and CI system for numpy is deployed and how to enable support for a new platform like win/arm64.
For CI, it should all be publicly available (free for open source) CI. We use GitHub Actions and Azure DevOps for the heavy lifting, plus CircleCI (doc builds), and TravisCI (arm64, ppc64le, s390x).
One of the main issues in supporting win/arm64 build would be due to the lack of win/arm64 VMs available on the cloud. But I see we have been producing binary wheels for Apple M1 platforms on pypi and conda repository for some time which also lacks the cloud VM support. I think we could take some learnings from Apple M1 support and look at how a similar strategy can be used for win/arm64.
Hi Niyas, thanks for asking. The learnings are mainly that without regular CI resources being available, this is a huge pain (it's been a year since hardware became available, and support is still spotty). Apple M1 is a bit of a special case, because many users and contributors, and even a few maintainers, now have that hardware locally. On the other hand, Windows on arm64 is not really a thing yet that a significant amount of people have. So probably our stance is that we'll accept patches for it, but cannot think about custom CI build agents (too much maintenance overhead).
I would like to hear if anyone has any thoughts on this topic. Also, any pointers to understand numpy wheel generation and CI flow for similar platforms would be helpful as well.
Our current wheel build machinery is at https://github.com/MacPython/numpy-wheels/, but please ignore that. We just merged a PR which uses cibuildwheel into the main repo. That should be the target. If cibuildwheel has/gains the ability to produce win-arm64 wheels, that'd be a good first step towards support. More generally: it'd be good to look at support in conda-forge, that had across-the-board macOS M1 support fairly soon, and is in general a much more capable setup for supporting new hardware like this. Cheers, Ralf
<snip> On Tue, Nov 2, 2021 at 1:07 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
Our current wheel build machinery is at https://github.com/MacPython/numpy-wheels/, but please ignore that. We just merged a PR which uses cibuildwheel into the main repo. That should be the target. If cibuildwheel has/gains the ability to produce win-arm64 wheels, that'd be a good first step towards support.
That would be https://github.com/numpy/numpy/pull/20102 I presume? I expect quite a few projects using numpy will follow your lead here. Thanks! Peter
On Tue, Nov 2, 2021 at 3:15 PM Peter Cock <p.j.a.cock@googlemail.com> wrote:
<snip>
On Tue, Nov 2, 2021 at 1:07 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
Our current wheel build machinery is at
but please ignore that. We just merged a PR which uses cibuildwheel into
That should be the target. If cibuildwheel has/gains the ability to
https://github.com/MacPython/numpy-wheels/, the main repo. produce win-arm64 wheels,
that'd be a good first step towards support.
That would be https://github.com/numpy/numpy/pull/20102 I presume?
Yes indeed. I expect quite a few projects using numpy will follow your lead here.
Thanks!
It's not quite our lead - quite a few other projects have been using cibuildwheel already for the last year or longer. It's just continued to improve, and it is clear now that cibuildwheel is the best supported way to build wheels. Cheers, Ralf
Hello, It has been a while since we discussed adding CI/CD support for Windows on Arm. Let me share a few updates and see if we can find a way forward. First of all, Thanks for suggesting cibuildwheel. We have couple of patches in progress to add native [1] and cross-compilation support [2] to cibuildwheel. They could solve the binary wheel creation and releases. However, we may still lack support for CI. I would like to check if there is any interest in using Windows/Arm64 VMs in Azure [3] or Volterra machines ( new Windows/Arm64 Dev Boxes ) [4] to add CI support for WoA platform. In order to use Azure VMs with GitHub Actions or Azure Pipeline, we must set up self-hosted runners. I am not sure if that would be an acceptable workflow for the numpy community. If that could work we could potentially find ways to fund the required VMs for NumPy project. Also, we could request access to new Volterra machines for Numpy CI/CD. I know it is not easy to maintain them as part of CI/CD workflow, but if any maintainers are interested in using them for CI/CD purposes, we can progress on that discussion. Let me know what you think. Thanks Niyas [1] https://github.com/pypa/cibuildwheel/pull/920 [2] https://github.com/pypa/cibuildwheel/pull/1108 [3] https://azure.microsoft.com/en-us/blog/now-in-preview-azure-virtual-machines... [4] https://blogs.windows.com/windowsdeveloper/2022/05/24/create-next-generation... On Tue, 2 Nov 2021 at 14:33, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Tue, Nov 2, 2021 at 3:15 PM Peter Cock <p.j.a.cock@googlemail.com> wrote:
<snip>
On Tue, Nov 2, 2021 at 1:07 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
Our current wheel build machinery is at
but please ignore that. We just merged a PR which uses cibuildwheel into the main repo. That should be the target. If cibuildwheel has/gains the ability to
https://github.com/MacPython/numpy-wheels/, produce win-arm64 wheels,
that'd be a good first step towards support.
That would be https://github.com/numpy/numpy/pull/20102 I presume?
Yes indeed.
I expect quite a few projects using numpy will follow your lead here.
Thanks!
It's not quite our lead - quite a few other projects have been using cibuildwheel already for the last year or longer. It's just continued to improve, and it is clear now that cibuildwheel is the best supported way to build wheels.
Cheers, Ralf
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: niyas.sait@linaro.org
When you say "we could request access to new Volterra machines": how do you see that playing out? Who would provide them, and who would maintain them? Would they be available to the more general scientific python community? Who would pay the bill? If the people supporting this niche platform stop doing so, can we drop the wheels? CPython has PEP 11 [0], maybe this effort should start by writing a similar NEP for the scientific python community. Matti [0] https://peps.python.org/pep-0011 On 6/6/22 13:35, Niyas Sait wrote:
Hello,
It has been a while since we discussed adding CI/CD support for Windows on Arm. Let me share a few updates and see if we can find a way forward.
First of all, Thanks for suggesting cibuildwheel. We have couple of patches in progress to add native [1] and cross-compilation support [2] to cibuildwheel. They could solve the binary wheel creation and releases. However, we may still lack support for CI.
I would like to check if there is any interest in using Windows/Arm64 VMs in Azure [3] or Volterra machines ( new Windows/Arm64 Dev Boxes ) [4] to add CI support for WoA platform.
In order to use Azure VMs with GitHub Actions or Azure Pipeline, we must set up self-hosted runners. I am not sure if that would be an acceptable workflow for the numpy community. If that could work we could potentially find ways to fund the required VMs for NumPy project.
Also, we could request access to new Volterra machines for Numpy CI/CD. I know it is not easy to maintain them as part of CI/CD workflow, but if any maintainers are interested in using them for CI/CD purposes, we can progress on that discussion.
Let me know what you think.
Thanks Niyas
[1] https://github.com/pypa/cibuildwheel/pull/920 [2] https://github.com/pypa/cibuildwheel/pull/1108 [3] https://azure.microsoft.com/en-us/blog/now-in-preview-azure-virtual-machines... [4] https://blogs.windows.com/windowsdeveloper/2022/05/24/create-next-generation...
When you say "we could request access to new Volterra machines": how do you see that playing out? Who would provide them, and who would maintain them? Would they be available to the more general scientific python community? Who would pay the bill?
Microsoft, Qualcomm, and Arm are very interested in native support for Numpy and other scientific packages for Windows on Arm platform. And we have contacts with them to start the relevant discussion if there is interest to use those dev boxes (local or remote access) or Azure VMs for CI/CD purposes. Hope that answers your questions. Niyas On Mon, 6 Jun 2022 at 14:51, Matti Picus <matti.picus@gmail.com> wrote:
When you say "we could request access to new Volterra machines": how do you see that playing out? Who would provide them, and who would maintain them? Would they be available to the more general scientific python community? Who would pay the bill? If the people supporting this niche platform stop doing so, can we drop the wheels? CPython has PEP 11 [0], maybe this effort should start by writing a similar NEP for the scientific python community.
Matti
[0] https://peps.python.org/pep-0011
On 6/6/22 13:35, Niyas Sait wrote:
Hello,
It has been a while since we discussed adding CI/CD support for Windows on Arm. Let me share a few updates and see if we can find a way forward.
First of all, Thanks for suggesting cibuildwheel. We have couple of patches in progress to add native [1] and cross-compilation support [2] to cibuildwheel. They could solve the binary wheel creation and releases. However, we may still lack support for CI.
I would like to check if there is any interest in using Windows/Arm64 VMs in Azure [3] or Volterra machines ( new Windows/Arm64 Dev Boxes ) [4] to add CI support for WoA platform.
In order to use Azure VMs with GitHub Actions or Azure Pipeline, we must set up self-hosted runners. I am not sure if that would be an acceptable workflow for the numpy community. If that could work we could potentially find ways to fund the required VMs for NumPy project.
Also, we could request access to new Volterra machines for Numpy CI/CD. I know it is not easy to maintain them as part of CI/CD workflow, but if any maintainers are interested in using them for CI/CD purposes, we can progress on that discussion.
Let me know what you think.
Thanks Niyas
[1] https://github.com/pypa/cibuildwheel/pull/920 [2] https://github.com/pypa/cibuildwheel/pull/1108 [3]
https://azure.microsoft.com/en-us/blog/now-in-preview-azure-virtual-machines...
[4]
https://blogs.windows.com/windowsdeveloper/2022/05/24/create-next-generation... _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: niyas.sait@linaro.org
On Mon, Jun 6, 2022 at 5:08 PM Niyas Sait <niyas.sait@linaro.org> wrote:
When you say "we could request access to new Volterra machines": how do you see that playing out? Who would provide them, and who would maintain them? Would they be available to the more general scientific python community? Who would pay the bill?
Microsoft, Qualcomm, and Arm are very interested in native support for Numpy and other scientific packages for Windows on Arm platform. And we have contacts with them to start the relevant discussion if there is interest to use those dev boxes (local or remote access) or Azure VMs for CI/CD purposes.
Hi Niyas, I'd be interested in remote access to a development machine. This will help with NumPy and SciPy; and it would allow testing binaries resulting from a cross-compilation step. I don't have a preference for a Volterra box or an Azure VM - as long as it can be used for development purposes the details of getting access are not that important I'd say.
Hope that answers your questions.
Niyas
On Mon, 6 Jun 2022 at 14:51, Matti Picus <matti.picus@gmail.com> wrote:
When you say "we could request access to new Volterra machines": how do you see that playing out? Who would provide them, and who would maintain them? Would they be available to the more general scientific python community? Who would pay the bill? If the people supporting this niche platform stop doing so, can we drop the wheels? CPython has PEP 11 [0], maybe this effort should start by writing a similar NEP for the scientific python community.
Matti
[0] https://peps.python.org/pep-0011
On 6/6/22 13:35, Niyas Sait wrote:
Hello,
It has been a while since we discussed adding CI/CD support for Windows on Arm. Let me share a few updates and see if we can find a way forward.
First of all, Thanks for suggesting cibuildwheel. We have couple of patches in progress to add native [1] and cross-compilation support [2] to cibuildwheel. They could solve the binary wheel creation and releases. However, we may still lack support for CI.
I would like to check if there is any interest in using Windows/Arm64 VMs in Azure [3] or Volterra machines ( new Windows/Arm64 Dev Boxes ) [4] to add CI support for WoA platform.
In order to use Azure VMs with GitHub Actions or Azure Pipeline, we must set up self-hosted runners. I am not sure if that would be an acceptable workflow for the numpy community. If that could work we could potentially find ways to fund the required VMs for NumPy project.
I don't think integration in the main NumPy CI is an option right now, however I also don't think it is necessary. A standalone daily or weekly scheduled job that would notify interested devs should already be very helpful. Cheers, Ralf
Also, we could request access to new Volterra machines for Numpy CI/CD. I know it is not easy to maintain them as part of CI/CD workflow, but if any maintainers are interested in using them for CI/CD purposes, we can progress on that discussion.
Let me know what you think.
Thanks Niyas
[1] https://github.com/pypa/cibuildwheel/pull/920 [2] https://github.com/pypa/cibuildwheel/pull/1108 [3]
https://azure.microsoft.com/en-us/blog/now-in-preview-azure-virtual-machines...
[4]
https://blogs.windows.com/windowsdeveloper/2022/05/24/create-next-generation... _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: niyas.sait@linaro.org
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ralf.gommers@googlemail.com
Hi Niyas, I'd be interested in remote access to a development machine. This will help with NumPy and SciPy; and it would allow testing binaries resulting from a cross-compilation step. I don't have a preference for a Volterra box or an Azure VM - as long as it can be used for development purposes the details of getting access are not that important I'd say.
Thanks, Ralf for the interest. I will sort it out and will let you know. I don't think integration in the main NumPy CI is an option right now,
however I also don't think it is necessary. A standalone daily or weekly scheduled job that would notify interested devs should already be very helpful.
FYI, Linaro has set up a downstream CI (Nightly) to track NumPy. See log for one of the previous runs here - https://gitlab.com/Linaro/windowsonarm/nightly/-/jobs/2622310764. Hopefully, that will help. Niyas On Tue, 21 Jun 2022 at 13:05, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Mon, Jun 6, 2022 at 5:08 PM Niyas Sait <niyas.sait@linaro.org> wrote:
When you say "we could request access to new Volterra machines": how do you see that playing out? Who would provide them, and who would maintain them? Would they be available to the more general scientific python community? Who would pay the bill?
Microsoft, Qualcomm, and Arm are very interested in native support for Numpy and other scientific packages for Windows on Arm platform. And we have contacts with them to start the relevant discussion if there is interest to use those dev boxes (local or remote access) or Azure VMs for CI/CD purposes.
Hi Niyas, I'd be interested in remote access to a development machine. This will help with NumPy and SciPy; and it would allow testing binaries resulting from a cross-compilation step.
I don't have a preference for a Volterra box or an Azure VM - as long as it can be used for development purposes the details of getting access are not that important I'd say.
Hope that answers your questions.
Niyas
On Mon, 6 Jun 2022 at 14:51, Matti Picus <matti.picus@gmail.com> wrote:
When you say "we could request access to new Volterra machines": how do you see that playing out? Who would provide them, and who would maintain them? Would they be available to the more general scientific python community? Who would pay the bill? If the people supporting this niche platform stop doing so, can we drop the wheels? CPython has PEP 11 [0], maybe this effort should start by writing a similar NEP for the scientific python community.
Matti
[0] https://peps.python.org/pep-0011
On 6/6/22 13:35, Niyas Sait wrote:
Hello,
It has been a while since we discussed adding CI/CD support for Windows on Arm. Let me share a few updates and see if we can find a way forward.
First of all, Thanks for suggesting cibuildwheel. We have couple of patches in progress to add native [1] and cross-compilation support [2] to cibuildwheel. They could solve the binary wheel creation and releases. However, we may still lack support for CI.
I would like to check if there is any interest in using Windows/Arm64 VMs in Azure [3] or Volterra machines ( new Windows/Arm64 Dev Boxes ) [4] to add CI support for WoA platform.
In order to use Azure VMs with GitHub Actions or Azure Pipeline, we must set up self-hosted runners. I am not sure if that would be an acceptable workflow for the numpy community. If that could work we could potentially find ways to fund the required VMs for NumPy project.
I don't think integration in the main NumPy CI is an option right now, however I also don't think it is necessary. A standalone daily or weekly scheduled job that would notify interested devs should already be very helpful.
Cheers, Ralf
Also, we could request access to new Volterra machines for Numpy CI/CD. I know it is not easy to maintain them as part of CI/CD workflow, but if any maintainers are interested in using them for CI/CD purposes, we can progress on that discussion.
Let me know what you think.
Thanks Niyas
[1] https://github.com/pypa/cibuildwheel/pull/920 [2] https://github.com/pypa/cibuildwheel/pull/1108 [3]
https://azure.microsoft.com/en-us/blog/now-in-preview-azure-virtual-machines...
[4]
https://blogs.windows.com/windowsdeveloper/2022/05/24/create-next-generation... _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: niyas.sait@linaro.org
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ralf.gommers@googlemail.com
_______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: niyas.sait@linaro.org
On Wed, Jun 22, 2022 at 11:35 AM Niyas Sait <niyas.sait@linaro.org> wrote:
Hi Niyas, I'd be interested in remote access to a development machine.
This will help with NumPy and SciPy; and it would allow testing binaries resulting from a cross-compilation step. I don't have a preference for a Volterra box or an Azure VM - as long as it can be used for development purposes the details of getting access are not that important I'd say.
Thanks, Ralf for the interest. I will sort it out and will let you know.
Thanks Niyas
I don't think integration in the main NumPy CI is an option right now,
however I also don't think it is necessary. A standalone daily or weekly scheduled job that would notify interested devs should already be very helpful.
FYI, Linaro has set up a downstream CI (Nightly) to track NumPy. See log for one of the previous runs here - https://gitlab.com/Linaro/windowsonarm/nightly/-/jobs/2622310764. Hopefully, that will help.
Nice! I see you have Matplotlib and Pillow as well in https://gitlab.com/Linaro/windowsonarm/packages. In case you plan to add SciPy, please feel free to ping me there (Gitlab username ralfgommers, GitHub rgommers). I have also added the Windows on ARM use case to our cross-compiling tracker issue: https://github.com/scipy/scipy/issues/14812. Cheers, Ralf
participants (4)
-
Matti Picus -
Niyas Sait -
Peter Cock -
Ralf Gommers