From: Nick Coghlan [mailto:ncoghlan@gmail.com] Sent: quinta-feira, 23 de março de 2017 23:59 To: Leonardo Bianconi Cc: distutils-sig@python.org Subject: Re: [Distutils] Wheel files for PPC64le On 24 March 2017 at 05:00, Leonardo Bianconi mailto:leonardo.bianconi@eldorado.org.br> wrote: Hi all! I have been discussed the creation of a PEP, that describes how to create wheel files for the PPC64le architecture on wheel-builders (https://mail.python.org/pipermail/wheel-builders/) since January (https://mail.python.org/pipermail/wheel-builders/2017-January/000245.html). Thanks Leonardo! As all discussion from that list are done, the next step is it be reviewed here, and then create a draft on github, You can submit the PR to the PEPs repo whenever you're ready - it's actually handy to have the PEP number assigned fairly early as a convenient reference for the proposal. The ``manylinux3`` policy ========================= Based on PEP 513 [1]_, the policy follows the same rules and library dependencies, but with the following versions for backward compatibility and base Operational System: * Backward compatibility versions: GLIBC <= 2.17 CXXABI <= 1.3.7 GLIBCXX <= 3.4.9 GCC <= 4.8.5 * Base Operational System: The stock O.S. release need to be the CentOS 7 [4]_, as it is the first CentOS release available for PowerPC64 little endian. The tag version for ppc64le architecture starts with 3 (``manylinux3_``), as it is supposed to be the version to match the CentOS 7 [4]_ in the future of the tag for x86_64/i686 architecture. There is the possibility of both tags diverge until it reaches the version 3, then a new PEP may be create to converge both to the manylinux baseline. Having manylinuxN consistently align with CentOS(N+4) seems reasonable to me for simplicity's sake, but there should be a discussion in the PEP around how that aligns with ppc64le support on other LTS distros (mainly Debian and Ubuntu). Given the relative dates involved, I'd expect manylinux-style binaries compiled on CentOS 7 to also work on Ubuntu 14.04, 16.04 and Debian 8, but the PEP should explicitly confirm that the nominated symbol versions above are available on all of those distros. Ok, I can add it to the PEP, but regarding the supported distros, the older than CentOS 7 may not be compatible, based on the backward compatibility rules, that does not guarantee compatibility with older versions, only with newer. I sent a message about it here https://mail.python.org/pipermail/wheel-builders/2017-March/000265.html. Compilation of Compliant Wheels =============================== As compiling wheel files that meet the ``manylinux3`` standard for PowerPC64 little endian requires a specific Linux distro and version, the following tool is provided: Docker Image (Will be implemented when CentOS be available on Docker) ------------ The Docker Image is based on CentOS 7 [4]_, which is the first PowerPC 64 little endian CentOS release. The Image contains all necessary tools in the requested version to build wheel files (gcc, g++ and gfortran 4.8.5). These seem to be present now: https://hub.docker.com/r/ppc64le/centos/tags/ I'm not clear on the provenance of the 'ppc64le' user account though, so I've asked for clarification: ttps://twitter.com/ncoghlan_dev/status/845099237117329408http://twitter.com/ncoghlan_dev/status/845099237117329408 Platform detection for Installers ================================= The platform detection is almost the same as described in PEP 513 [1]_, but with the following proposed change: 1. Add the platform ppc64le in the platform list as a compatible one: [``linux-x86_64``, ``linux-i686``, ``linux-ppc64le``] 2. Add an if to switch architecture and consider the correct version of the GLIBC on ``return have_compatible_glibc(2, 5)``. I don't think is quite that simple, as installers need to be able to figure out: - on manylinux3 compatible platforms, prefer manylinux3 to manylinux1 - on manylinux3 *in*compatible platforms, only consider manylinux1 And that means asking the question: when combined with the option of the distro-provided `_manylinux` module, is "have_compatible_glibc(2, 5) and not have_compatible_glibc(2, 17)" an adequate check for the latter case? (My inclination is to say "yes", but it would be helpful to have some more concrete data on glibc versions in different distros of interest) Well, I didn’t realize that proposing a new tag would require an additional check about the tags, which will be a requirement for the manylinux2 as well, when the CentOS 5 be replaced by CentOS 6 for x86_64/i686. I need to check where and how the method “is_manylinux1_compatible” is used to think how it would be done. I will check that and propose how to do it. Beyond that, I think the main open question would be: do we go ahead and define the full `manylinux3` specification now? CentOS 7+, Ubuntu 14.04+, Debian 8+ compatibility still covers a *lot* of distros and deployments, and doing so means folks can bring the latest versions of gcc to bear on their code, rather than being limited to the last version that was made available for RHEL/CentOS 5 (gcc 4.8). Actually the idea was make it available for PPC64le, just as it is available to x86_64/i686 nowadays, like porting it. I didn’t think about the definition of all requirements for the manylinux3 for all architectures, as it can change until x86_64/i686 get to the manylinux3. Being limited to an old version, as CentOS 5 (gcc 4.8) is a requirement from PEP 513, which guarantees the backward compatibility, right? I do not want to change it, this proposal is just to create a tag for PPC64le, until both architectures get to the same base distro version. As I said above, I have already sent a message about basing it on CentOS 7, which does not guarantee the compatibility with older distros (example: Ubuntu 14.04). Is there any thinking about base on a newer distro and make the wheel files compatible with distros older than it? Sorry if I’m missing something here. Going down that path would also means things would be simpler on the PyPI front - it could just allow manylinux3 for any architecture and let installers decide whether or not to use them. Cheers, Nick. I’m coping the Bruno Rosa, which will be involved with this PEP as well. Cheers, Leonardo Bianconi. -- Nick Coghlan | ncoghlan@gmail.commailto:ncoghlan@gmail.com | Brisbane, Australia

Hi there, First of all, thanks for checking out the information about Docker images, Nick! Since Leonardo's last email got some formatting issues, I'm fixing it (mostly manually) and sending it here again. Kind regards, Bruno Rosa --------------------------------------------------

Having manylinuxN consistently align with CentOS(N+4) seems reasonable to me for simplicity's sake, but there should be a discussion in the PEP around how that aligns with ppc64le support on other LTS distros (mainly Debian and Ubuntu). Given the relative dates involved, I'd expect manylinux-style binaries compiled on CentOS 7 to also work on Ubuntu 14.04, 16.04 and Debian 8, but the PEP should explicitly confirm that the nominated symbol versions above are available on all of those distros.

Ok, I can add it to the PEP, but regarding the supported distros, the older than CentOS 7 may not be compatible, based on the backward compatibility rules, that does not guarantee compatibility with older versions, only with newer. I sent a message about it here https://mail.python.org/pipermail/wheel-builders/2017-March/000265.html.

I don't think is quite that simple, as installers need to be able to figure out: - on manylinux3 compatible platforms, prefer manylinux3 to manylinux1 - on manylinux3 *in*compatible platforms, only consider manylinux1 And that means asking the question: when combined with the option of the distro-provided `_manylinux` module, is "have_compatible_glibc(2, 5) and not have_compatible_glibc(2, 17)" an adequate check for the latter case? (My inclination is to say "yes", but it would be helpful to have some more concrete data on glibc versions in different distros of interest)

Well, I didn’t realize that proposing a new tag would require an additional check about the tags, which will be a requirement for the manylinux2 as well, when the CentOS 5 be replaced by CentOS 6 for x86_64/i686. I need to check where and how the method “is_manylinux1_compatible” is used to think how it would be done. I will check that and propose how to do it.

Beyond that, I think the main open question would be: do we go ahead and define the full `manylinux3` specification now? CentOS 7+, Ubuntu 14.04+, Debian 8+ compatibility still covers a *lot* of distros and deployments, and doing so means folks can bring the latest versions of gcc to bear on their code, rather than being limited to the last version that was made available for RHEL/CentOS 5 (gcc 4.8).

Actually the idea was make it available for PPC64le, just as it is available to x86_64/i686 nowadays, like porting it. I didn’t think about the definition of all requirements for the manylinux3 for all architectures, as it can change until x86_64/i686 get to the manylinux3. Being limited to an old version, as CentOS 5 (gcc 4.8) is a requirement from PEP 513, which guarantees the backward compatibility, right? I do not want to change it, this proposal is just to create a tag for PPC64le, until both architectures get to the same base distro version. As I said above, I have already sent a message about basing it on CentOS 7, which does not guarantee the compatibility with older distros (example: Ubuntu 14.04). Is there any thinking about base on a newer distro and make the wheel files compatible with distros older than it? Sorry if I’m missing something here. I’m coping the Bruno Rosa, which will be involved with this PEP as well. Cheers, Leonardo Bianconi. --------------------------------------------------