Hi Nathaniel, Thanks for your answer. Basing on your example of RHEL and Ubuntu, let's take RHEL 6 which uses glibc 2.12. If you cross-compile for it (using the same gcc RHEL uses), wheel surely will work on Ubuntu 18.10 :) I think it's not of an issue, since wheels are built with these minimal runtime requirements anyway, unless they're built on a local machine – but in this case they will just work™; anyway, having a working toolchain is not the scope of Python tooling. Gentoo has an awesome crossdev tool for creating cross-toolchains, and there's crosstool-ng of course. It looks like there are workarounds for putting code built on newer systems to older ones though, but they seem to be pretty tedious ([1]) Speaking of runtime detection, I see it this way for example – since one of the most reliable ways to check for program's dependencies is invoking something like ldd or objdump, it can essentially be done the same way: 1. Pick the minimal required code to extract ELF's ".interp" field [2] (I used code from [3]); 2. Process sys.executable with it; Here what it returns (grepping by "/lib" because it starts from a new line): Gentoo amd64 glibc # python3 readelf.py $(which python3) | grep "/lib" b'/lib64/ld-linux-x86-64.so.2\x00' Alpine amd64 docker (official python3 alpine image): # python3 readelf.py $(which python3) | grep "/lib" b'/lib/ld-musl-x86_64.so.1\x00' 3. Essentially that's enough in my opinion, but we can go further and do what ldd does: # /lib/ld-musl-x86_64.so.1 --list $(which python3) /lib/ld-musl-x86_64.so.1 (0x7fbc36567000) libpython3.7m.so.1.0 => /usr/local/lib/libpython3.7m.so.1.0 (0x7fbc3622a000) libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fbc36567000) Basically it's just string matching here, and the only question now if the name of dynamic linker is enough or all libs should be iterated until perfect "musl" or "libc" match. Parsing "Dynamic section" turns out to be pretty useless – it's empty on Alpine (or parsing code is buggy). If ".interp" field is not available, then interpreter is statically linked :) 4. If any glibc-specific functionality is needed at this point, code from PEP 513 is really good. Maybe it's also better to put it first and use ELF parsing if it failed to open glibc in the first place. Thanks, Alex [1] https://snorfalorpagus.net/blog/2016/07/17/compiling-python-extensions-for-o... [2] https://www.linuxjournal.com/article/1060 [3] https://github.com/detailyang/readelf/blob/master/readelf/readelf.py#L545 On Wed, Feb 20, 2019 at 1:50 AM Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Feb 19, 2019 at 3:28 PM Alexander Revin <lyssdod@gmail.com> wrote:
Hi all,
I have an idea regarding Python binary wheels on non-glibc platforms, and it seems that initially I've posted it to the wrong list ([1])
Long story short, the proposal is to use platform tuples (like compiler ones) for wheel names, which will allow much broader platform support, for example:
package-1.0-cp36-cp36m-amd64_linux_gnu.whl package-1.0-cp36-cp36m-amd64_linux_musl.whl
So eventually only {platform tag} part will be modified. Glibc/musl detection is quite trivial and eventually will be based on existing one in PEP 513 [2].
The challenge here is: the purpose of a target triple is to tell a compiler/linker toolchain which kind of code they should generate, e.g. when cross-compiling. The purpose of a wheel tag is to tell you whether a given wheel will work on a given system. It turns out these are different things :-).
For example, Ubuntu 18.10 and RHEL 6 are both 'amd64-linux-gnu', because they use the same instruction set, the same binary format (ELF), etc. But if you build a wheel on Ubuntu 18.10, it definitely will not work on RHEL 6. (The other way around might work, if you do other things right.)
In practice Windows and macOS are already fine; the place where this would be useful is Linux wheels for platforms that use non-Intel-based architectures or non-glibc-libcs. We do have an idea for making it easier to support newer glibcs and also extending to all architectures: https://mail.python.org/archives/list/distutils-sig@python.org/thread/6AFS4H...
Adding musl is a bit trickier since I'm not sure what the details of their ABI compatibility are, and they intentionally make it difficult to figure out whether you're running on musl. But if someone could convince them to publish more information then we could fix that too.
-n
-- Nathaniel J. Smith -- https://vorpus.org