NumPy cross-compilation on macOS host
Hi, I spent some time trying to cross-compile NumPy for Linux ARM64 on a macOS host as part of an OpenWRT compilation [1]. It was not so bad but certainly not trivial and I was wondering if we could use my experience to help streamline the process. I encountered two challenges: 1. Python distutils did not allow me to provide a proper cross-ranlib program path. I opened an issue for this on the Python bug tracker [2]. 2. numpy distutils scripts check sys.platform in many places. sys.platform returns the platform on which compilation occurs (darwin in my case) which in case of cross-compilation is not the one on which the code will be run (linux). This of course results in invalid compiler and linker flags. I made a quick'n'dirty fix [3] and got numpy to compile and run correctly (but I have yet to run the full test suite). Of course this is not a proper way to solve this so I was hoping to get some feedback on the preferred/best way. I did not find any info about an official way to do this with distutils. One idea (from @commodo on GitHub) would be to just change my hardcoded string ("Linux") to something like this: target_platform = os.getenviron(“TARGET_PLATFORM”, sys.platform) It could also be further refactored by moving it to a single place but while certainly more elegant and "proper" it may ultimately make it less readable and harder to follow. I am also unsure where would be the best place to put it and how to pass it around to other modules. [1]: https://github.com/openwrt/packages/pull/9797 [2]: https://bugs.python.org/issue37916 [3]: https://github.com/openwrt/packages/pull/9797/commits/e701e122fb9c78d5cecffb...
On Fri, Aug 23, 2019 at 2:28 AM Jakub Piotr Cłapa <jpc-ml@zenburn.net> wrote:
Hi,
I spent some time trying to cross-compile NumPy for Linux ARM64 on a macOS host as part of an OpenWRT compilation [1]. It was not so bad but certainly not trivial and I was wondering if we could use my experience to help streamline the process.
Thanks, that would be great! We have quite some issues with cross-compilation, and usually they're hard to resolve (plus it's not something we officially support - but there's a clear need for improvement). Note that conda is also a cross-compiler (to the same system), sometimes worth looking into how it deals with (numpy.)distutils issues.
I encountered two challenges:
1. Python distutils did not allow me to provide a proper cross-ranlib program path. I opened an issue for this on the Python bug tracker [2].
That seems like a straightforward enough patch. distutils isn't well-maintained but hopefully that gets merged soon. If it doesn't get responses in a couple of weeks, please feel free to ping me and I can review/test (I'm not a core Python dev, but it may help push it forwards).
2. numpy distutils scripts check sys.platform in many places. sys.platform returns the platform on which compilation occurs (darwin in my case) which in case of cross-compilation is not the one on which the code will be run (linux). This of course results in invalid compiler and linker flags.
I made a quick'n'dirty fix [3] and got numpy to compile and run correctly (but I have yet to run the full test suite). Of course this is not a proper way to solve this so I was hoping to get some feedback on the preferred/best way. I did not find any info about an official way to do this with distutils.
One idea (from @commodo on GitHub) would be to just change my hardcoded string ("Linux") to something like this:
target_platform = os.getenviron(“TARGET_PLATFORM”, sys.platform)
That seems reasonable enough. Prefixing that name with NPY_ would be good, to not get accidental clashes. I'm not sure if there's a more standard/preferred way, I haven't come across it in distutils. There's some issues with reasonably detailed discussion for SciPy - this one may be useful: https://github.com/scipy/scipy/issues/8571
It could also be further refactored by moving it to a single place but while certainly more elegant and "proper" it may ultimately make it less readable and harder to follow. I am also unsure where would be the best place to put it and how to pass it around to other modules.
Keeping all platform comparisons where they are now seems right. Just need one central place to parse NPY_TARGET_PLATFORM - perhaps in numpy/distutils/misc_util.py Cheers, Ralf
[1]: https://github.com/openwrt/packages/pull/9797 [2]: https://bugs.python.org/issue37916 [3]:
https://github.com/openwrt/packages/pull/9797/commits/e701e122fb9c78d5cecffb... _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
participants (2)
-
Jakub Piotr Cłapa
-
Ralf Gommers