Mailman 3 distutils.util.get_host_platform() and AIX - Distutils-SIG

distutils.util.get_host_platform() and AIX

older
It all works - wheels for AIX - up...

Michael

1 Aug 2019 1 Aug '19

2:37 p.m.

Recently I have become more interested in looking at how to simplify packaging for AIX. Currently I more or less ignore anything pip and/or distutils could mean for a python "user" (e.g., AIX sys admin and/or Python developer working on AIX) as I just run "pip install XYZ" locally, find what constitutes the equivalent of a bdist, and repackage those as 'installp' packages. For someone with root access this is ok - as installp requires root (or RBAC equivalent) to install. As I would like to make installing modules more Python like - I asked around and was pointed at piwheels (which looks very useful for my aspiration) but also at PEP425 and the platform-tag. There are several things I need to learn about dist-utils, and how packaging tagging needs to be considered when, e.g., Python is built on AIX 7.1 and the module is built on AIX 7.2 - there may be AIX ABI differences that would prevent the module built on AIX 7.2 from working properly on AIX 7.1 while the same module built on AIX 7.1 will run with no issue on AIX 7.2. (A simplified explanation is that libc.a on AIX 7.2 knows how to work with applications linked against libc.a from earlier versions of AIX, but earlier versions of AIX do not know how to deal with libc.a from AIX 7.2.) Taken to an extreme: Python(2|3) and modules built on AIX 5.3 TL7 run, unaltered, on all levels of AIX including the latest AIX 7.2 - while my expectation is that executable and modules built on AIX 7.2 TL5 (latest level) might not run on AIX 7.2 TL4. In short, "version", "release" are in themselves not enough - the TL (technology level) should also be considered. Additionally, what I also see "missing" is the platform.architecture()[0] value. By default, this is still 32bit on AIX - but it is important - especially for pre-built eggs and/or wheels. In the AIX world - the OS-level name scheme is usually: VR00-TL-SP (Version, Release, TechnologyLevel, ServicePack). There is also a value compareable to a builddate, but for distutils purposes has no value (I can think of). So, to my question: currently, for AIX get_host_platform returns something such as: aix-6.1. Considering above: what would you - as more expericenced with multi-oslevel packaging and low levels are accepted by high-levels, but not v.v. "What should the AIX get_host_platform() string contain?" At a minimum I forsee: return "%s-%s.%s-%s" % (osname, version, release, platform.architecture()[0]) But this does not address potential issues where the TL level within a version.release has changed. (X.Y.TL5 built packages MIGHT work on X.YTL4, but there is no reason to expect them to. So, I would look to something that remains recognizable, but uses different 'punctuation' e.g., oslevel -s returns a string such as: 6100-09-10-1731 Then using the equivalent of: version, release, service, builddata = '6100-09-10-1731'.split('-') return "%s-%s.%s.%s-%s" % (osname, version, release, service, platform.architecture()[0]) Note: no special authority is needed to run "oslevel -s", but it does take time. So having a way, in the library to only have to actually call for the value once - is a great improvement. I can imagine a way to do it (store a static value somewhere, and when it is NULL aka not initialized call the program, otherwise return the value) - but "WHERE" - in distutils, or (ideally?) elsewhere in a more "central" library. Sincerely, Michael

Attachments:

signature.asc (application/pgp-signature — 488 bytes)

Show replies by date

Michael

2 Aug 2 Aug

12:32 p.m.

On 01/08/2019 11:07, Michael wrote:

...

So, to my question: currently, for AIX get_host_platform returns something such as: aix-6.1.

Considering above: what would you - as more expericenced with multi-oslevel packaging and low levels are accepted by high-levels, but not v.v. "What should the AIX get_host_platform() string contain?"

At a minimum I forsee: return "%s-%s.%s-%s" % (osname, version, release, platform.architecture()[0])

But this does not address potential issues where the TL level within a version.release has changed. (X.Y.TL5 built packages MIGHT work on X.YTL4, but there is no reason to expect them to.

So, I would look to something that remains recognizable, but uses different 'punctuation'

e.g., oslevel -s returns a string such as: 6100-09-10-1731

Then using the equivalent of:

version, release, service, builddata = '6100-09-10-1731'.split('-')

return "%s-%s.%s.%s-%s" % (osname, version, release, service, platform.architecture()[0])

What I forgot to mention - there is likely incompatibilities when different compilers are used. This is definetly the case when source files need a C compiler - and I fear that the different run-time environments of gcc versus xlc (which does not need/link to glibc). So, how does, e.g., macos account for differences between clang and gcc compiled executables and modules. Or are both compilers "gnu" oriented? Michael

Ronald Oussoren

3 Aug 3 Aug

3:46 p.m.

...

On 2 Aug 2019, at 09:02, Michael wrote:

On 01/08/2019 11:07, Michael wrote:

...
So, to my question: currently, for AIX get_host_platform returns something such as: aix-6.1.

Considering above: what would you - as more expericenced with multi-oslevel packaging and low levels are accepted by high-levels, but not v.v. "What should the AIX get_host_platform() string contain?"

At a minimum I forsee: return "%s-%s.%s-%s" % (osname, version, release, platform.architecture()[0])

But this does not address potential issues where the TL level within a version.release has changed. (X.Y.TL5 built packages MIGHT work on X.YTL4, but there is no reason to expect them to.

So, I would look to something that remains recognizable, but uses different 'punctuation'

e.g., oslevel -s returns a string such as: 6100-09-10-1731

Then using the equivalent of:

version, release, service, builddata = '6100-09-10-1731'.split('-')

return "%s-%s.%s.%s-%s" % (osname, version, release, service, platform.architecture()[0])

What I forgot to mention - there is likely incompatibilities when different compilers are used. This is definetly the case when source files need a C compiler - and I fear that the different run-time environments of gcc versus xlc (which does not need/link to glibc).

So, how does, e.g., macos account for differences between clang and gcc compiled executables and modules. Or are both compilers "gnu" oriented?

For macOS GCC and clang generate compatible code, that’s not a problem. For platform versions the situation is slightly more complicated, but it is in general possible to build on newer releases of macOS than you deploy on. However, this requires some support from the code you are building, such as not using APIs that aren’t available on the older version. That generally can cause issues with projects that use configure and can automatically pick up symbols available on the build platform. The Linux setup might be more interesting for you in that regard. The “manylinux” project [1] specifies how to build wheels that work on multiple linux versions, regardless of the version of libc. In short this works by building with an older libc, to ensure that the wheels only use symbol versions that are available on all supported linux versions. [1] https://github.com/pypa/manylinux https://github.com/pypa/manylinux Ronald

...

Michael

-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/OZX3X...

Michael

6 Aug 6 Aug

12:01 p.m.

I know I started a reply. I hate phones (for replies)... On 03/08/2019 12:16, Ronald Oussoren via Distutils-SIG wrote:

...

For macOS GCC and clang generate compatible code, that’s not a problem. Win! for macOS!

For platform versions the situation is slightly more complicated, but it is in general possible to build on newer releases of macOS than you deploy on. However, this requires some support from the code you are building, such as not using APIs that aren’t available on the older version. That generally can cause issues with projects that use configure and can automatically pick up symbols available on the build platform.

The Linux setup might be more interesting for you in that regard. The “manylinux” project [1] specifies how to build wheels that work on multiple linux versions, regardless of the version of libc. In short this works by building with an older libc, to ensure that the wheels only use symbol versions that are available on all supported linux versions.

To an extent - support for a "manylinux" approach is "built-in" to AIX. In practice, AIX has traditionally been 'friendly' to applications coming in binary form AIX built on older versions. There have been two releases where it was announced - there are potential issues (when AIX 5.0 and AIX 5.1 came - 64-bit applications had to be re-compiled. and AIX 5.3 TL6 to AIX 5.3TL7 - some applications "broke" - so back around 1999-2000 and 2005-2006 time frames. With the introduction of AIX 6.1 TL0 (and, in parallel AIX 5.3 TL7) IBM AIX offered, rather assured - binary compatibility when they linked against shared libraries (such as libc, aka - the "stuff" in /usr/lib). Also specified - an application linked against AIX 7.2 TL3 libraries will not run on AIX 7.1 or earlier versions of AIX - actually, assume it will not run on AIX 7.2 TL2 or earlier). Starting with Python 3.8 - sys.platform() will just say "aix", just as sys.platform() has just said "linux" since, iirc Python 3.2. (FYI). So, I'll read up on "manylinux" to see what I can lean on (read 'borrow'). I will be grateful for feedback!

...

[1] https://github.com/pypa/manylinux https://github.com/pypa/manylinux

Ronald

Michael

12:45 p.m.

On 03/08/2019 12:16, Ronald Oussoren via Distutils-SIG wrote:

...

On 01/08/2019 11:07, Michael wrote:

...
So, to my question: currently, for AIX get_host_platform returns something such as: aix-6.1.

Considering above: what would you - as more expericenced with multi-oslevel packaging and low levels are accepted by high-levels, but not v.v. "What should the AIX get_host_platform() string contain?"

At a minimum I forsee: return "%s-%s.%s-%s" % (osname, version, release, platform.architecture()[0])

But this does not address potential issues where the TL level within a version.release has changed. (X.Y.TL5 built packages MIGHT work on X.YTL4, but there is no reason to expect them to.

So, I would look to something that remains recognizable, but uses different 'punctuation'

e.g., oslevel -s returns a string such as: 6100-09-10-1731

Then using the equivalent of:

version, release, service, builddata = '6100-09-10-1731'.split('-')

return "%s-%s.%s.%s-%s" % (osname, version, release, service, platform.architecture()[0]) What I forgot to mention - there is likely incompatibilities when different compilers are used. This is definetly the case when source files need a C compiler - and I fear that the different run-time environments of gcc versus xlc (which does not need/link to glibc).

So, how does, e.g., macos account for differences between clang and gcc compiled executables and modules. Or are both compilers "gnu" oriented?

OK - after reading PEP513 (and PEP571) I see that the "approach" is different. These PEPs came up with a 'white-list' of acceptable shared libraries - while the AIX approach has focused on "new OS levels support old OS levels". In any case - I see this as a "format" for a wheel 'label' PKG-VERSION-cp27-cp27m-manylinux1_x86_64.whl where, I assume cp27 stands for CPython-2.7 - so the "front-end": PKG-VERSION-cp27-cp27m- is 'provided' and what I need to replace is the 'tail' manylinux1_x86_64.whl So, since my approach is NOT to have a list of acceptable libraries - I guess I would start with something like: PKG-VERSION-cp37-cp37m-aix_6100_07-32b.whl and PKG-VERSION-cp37-cp37md-aix_6100_07-32b.whl ++++ PEP571 mentions "try:\n import _manylinux\n..." As I look around I get the impression that this is a module that the manylinux docker image is 'inserting'. The deeper I look, the less I see in this approach. The "common-ground" is that they also state the 'heart' of having portable code is having access to an "old" machine - and providing that via docker. ++++ What I am looking to accomplish is to have a way to "accept" a package labeled PKG-VERSION-cp37-cp37md-aix_6100_09-32b.whl if the label "I" would generate is: PKG-VERSION-cp37-cp37md-aix_6100_10-32b.whl or PKG-VERSION-cp37-cp37md-aix_7100_07-32b.whl but not accept PKG-VERSION-cp37-cp37md-aix_6100_07-32b.whl or PKG-VERSION-cp37-cp37m-aix_6100_09-32b.whl or PKG-VERSION-cp37-cp37md-aix_6100_09-64b.whl This means the "mdu" characters need to match, also the cpXY - and the "32|64"b and the aix_XY00_TL needs to be lower, or equal to the 'running' system. So, what am I missing - and where should this (plus anything else) be inserted?

Michael

13 Aug 13 Aug

8:25 p.m.

On 06/08/2019 09:15, Michael wrote: OK - been about a week - so, before I forget everything a list-ping... Quick summary: re: manylinix1 approach: read several docs and checked CPython - please tell me if/where I am missing something. * I do not have a Linux system to look at separately, maybe I must look for a system (I can trash by accident) * the manylinux support is external to CPython (Lib/distutils) * how does pip/dist-utils interpet the "label" that is applied to a package * is the label applied by the CPython distutils package, or is that package "overwritten" by the distutils part of pip? This concerns whether I would need to write a PR for CPython, or for distutils (aka, who is the 'owner' of distutils I suppose there is more - but I hope to get some level of synchronization before getting too far along. Thanks for your assistance! Michael

...

...
On 01/08/2019 11:07, Michael wrote:

...
So, to my question: currently, for AIX get_host_platform returns something such as: aix-6.1.

Considering above: what would you - as more expericenced with multi-oslevel packaging and low levels are accepted by high-levels, but not v.v. "What should the AIX get_host_platform() string contain?"

At a minimum I forsee: return "%s-%s.%s-%s" % (osname, version, release, platform.architecture()[0])

But this does not address potential issues where the TL level within a version.release has changed. (X.Y.TL5 built packages MIGHT work on X.YTL4, but there is no reason to expect them to.

So, I would look to something that remains recognizable, but uses different 'punctuation'

e.g., oslevel -s returns a string such as: 6100-09-10-1731

Then using the equivalent of:

version, release, service, builddata = '6100-09-10-1731'.split('-')

return "%s-%s.%s.%s-%s" % (osname, version, release, service, platform.architecture()[0]) What I forgot to mention - there is likely incompatibilities when different compilers are used. This is definetly the case when source files need a C compiler - and I fear that the different run-time environments of gcc versus xlc (which does not need/link to glibc).

So, how does, e.g., macos account for differences between clang and gcc compiled executables and modules. Or are both compilers "gnu" oriented? OK - after reading PEP513 (and PEP571) I see that the "approach" is different. These PEPs came up with a 'white-list' of acceptable shared

On 03/08/2019 12:16, Ronald Oussoren via Distutils-SIG wrote: libraries - while the AIX approach has focused on "new OS levels support old OS levels".

In any case - I see this as a "format" for a wheel 'label'

PKG-VERSION-cp27-cp27m-manylinux1_x86_64.whl

where, I assume cp27 stands for CPython-2.7 - so the "front-end": PKG-VERSION-cp27-cp27m- is 'provided' and what I need to replace is the 'tail' manylinux1_x86_64.whl

So, since my approach is NOT to have a list of acceptable libraries - I guess I would start with something like:

PKG-VERSION-cp37-cp37m-aix_6100_07-32b.whl and PKG-VERSION-cp37-cp37md-aix_6100_07-32b.whl

++++ PEP571 mentions "try:\n import _manylinux\n..."

As I look around I get the impression that this is a module that the manylinux docker image is 'inserting'.

The deeper I look, the less I see in this approach. The "common-ground" is that they also state the 'heart' of having portable code is having access to an "old" machine - and providing that via docker.

++++ What I am looking to accomplish is to have a way to "accept" a package labeled

PKG-VERSION-cp37-cp37md-aix_6100_09-32b.whl if the label "I" would generate is:

PKG-VERSION-cp37-cp37md-aix_6100_10-32b.whl or PKG-VERSION-cp37-cp37md-aix_7100_07-32b.whl

but not accept

PKG-VERSION-cp37-cp37md-aix_6100_07-32b.whl or PKG-VERSION-cp37-cp37m-aix_6100_09-32b.whl or PKG-VERSION-cp37-cp37md-aix_6100_09-64b.whl

This means the "mdu" characters need to match, also the cpXY - and the "32|64"b and the aix_XY00_TL needs to be lower, or equal to the 'running' system.

So, what am I missing - and where should this (plus anything else) be inserted?

-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/Q4PGP...

Paul Moore

8:43 p.m.

A couple of quick points: * You want to focus on setuptools rather than distutils. Setuptools extends distutils with recent packaging developments, distutils is essentially just core Python legacy support these days. * Wheels (built binaries) are typically tagged as manylinux using the auditwheel project (https://github.com/pypa/auditwheel) although you are in theory allowed to manually apply the manylinux tag if you can verify that your wheel conforms to the requirements listed in the relevant manylinux spec/PEP. * By default, binaries built on linux are tagged as "linux" which means "no guarantee that this can be run on any machine other than the one it was built on". Such wheels cannot be uploaded to PyPI. * Pip includes rules that let it determine what tags are suitable for installation on this machine. The details are in https://packaging.python.org/specifications/platform-compatibility-tags/ (and that link references PEP 425 as well as the manylinux PEPs for the details). Whether manylinux is relevant for AIX is something you'll have to decide. Ultimately it depends on how well the default rules in PEP 425 manage to ensure that binaries are interoperable, which in turn probably depends on how Python typically gets built on AIX, and how compatible dynamically linked system libraries are across machines. Paul On Tue, 13 Aug 2019 at 15:56, Michael wrote:

...

On 06/08/2019 09:15, Michael wrote: OK - been about a week - so, before I forget everything a list-ping...

Quick summary: re: manylinix1 approach: read several docs and checked CPython - please tell me if/where I am missing something.

* I do not have a Linux system to look at separately, maybe I must look for a system (I can trash by accident) * the manylinux support is external to CPython (Lib/distutils) * how does pip/dist-utils interpet the "label" that is applied to a package * is the label applied by the CPython distutils package, or is that package "overwritten" by the distutils part of pip? This concerns whether I would need to write a PR for CPython, or for distutils (aka, who is the 'owner' of distutils

I suppose there is more - but I hope to get some level of synchronization before getting too far along.

Thanks for your assistance!

Michael

On 03/08/2019 12:16, Ronald Oussoren via Distutils-SIG wrote:

On 01/08/2019 11:07, Michael wrote:

So, to my question: currently, for AIX get_host_platform returns something such as: aix-6.1.

Considering above: what would you - as more expericenced with multi-oslevel packaging and low levels are accepted by high-levels, but not v.v. "What should the AIX get_host_platform() string contain?"

At a minimum I forsee: return "%s-%s.%s-%s" % (osname, version, release, platform.architecture()[0])

But this does not address potential issues where the TL level within a version.release has changed. (X.Y.TL5 built packages MIGHT work on X.YTL4, but there is no reason to expect them to.

So, I would look to something that remains recognizable, but uses different 'punctuation'

e.g., oslevel -s returns a string such as: 6100-09-10-1731

Then using the equivalent of:

version, release, service, builddata = '6100-09-10-1731'.split('-')

return "%s-%s.%s.%s-%s" % (osname, version, release, service, platform.architecture()[0])

What I forgot to mention - there is likely incompatibilities when different compilers are used. This is definetly the case when source files need a C compiler - and I fear that the different run-time environments of gcc versus xlc (which does not need/link to glibc).

So, how does, e.g., macos account for differences between clang and gcc compiled executables and modules. Or are both compilers "gnu" oriented?

OK - after reading PEP513 (and PEP571) I see that the "approach" is different. These PEPs came up with a 'white-list' of acceptable shared libraries - while the AIX approach has focused on "new OS levels support old OS levels".

In any case - I see this as a "format" for a wheel 'label'

PKG-VERSION-cp27-cp27m-manylinux1_x86_64.whl

where, I assume cp27 stands for CPython-2.7 - so the "front-end": PKG-VERSION-cp27-cp27m- is 'provided' and what I need to replace is the 'tail' manylinux1_x86_64.whl

So, since my approach is NOT to have a list of acceptable libraries - I guess I would start with something like:

PKG-VERSION-cp37-cp37m-aix_6100_07-32b.whl and PKG-VERSION-cp37-cp37md-aix_6100_07-32b.whl

++++ PEP571 mentions "try:\n import _manylinux\n..."

As I look around I get the impression that this is a module that the manylinux docker image is 'inserting'.

The deeper I look, the less I see in this approach. The "common-ground" is that they also state the 'heart' of having portable code is having access to an "old" machine - and providing that via docker.

++++ What I am looking to accomplish is to have a way to "accept" a package labeled

PKG-VERSION-cp37-cp37md-aix_6100_09-32b.whl if the label "I" would generate is:

PKG-VERSION-cp37-cp37md-aix_6100_10-32b.whl or PKG-VERSION-cp37-cp37md-aix_7100_07-32b.whl

but not accept

PKG-VERSION-cp37-cp37md-aix_6100_07-32b.whl or PKG-VERSION-cp37-cp37m-aix_6100_09-32b.whl or PKG-VERSION-cp37-cp37md-aix_6100_09-64b.whl

This means the "mdu" characters need to match, also the cpXY - and the "32|64"b and the aix_XY00_TL needs to be lower, or equal to the 'running' system.

So, what am I missing - and where should this (plus anything else) be inserted?

-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/Q4PGP...

-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/5JR7A...

Michael

9:22 p.m.

On 13/08/2019 17:13, Paul Moore wrote:

...

A couple of quick points: Many thanks for the pointers!

* You want to focus on setuptools rather than distutils. Setuptools extends distutils with recent packaging developments, distutils is essentially just core Python legacy support these days. * Wheels (built binaries) are typically tagged as manylinux using the auditwheel project (https://github.com/pypa/auditwheel) although you are in theory allowed to manually apply the manylinux tag if you can verify that your wheel conforms to the requirements listed in the relevant manylinux spec/PEP. * By default, binaries built on linux are tagged as "linux" which means "no guarantee that this can be run on any machine other than the one it was built on". Such wheels cannot be uploaded to PyPI. * Pip includes rules that let it determine what tags are suitable for installation on this machine. The details are in https://packaging.python.org/specifications/platform-compatibility-tags/ (and that link references PEP 425 as well as the manylinux PEPs for the details).

Whether manylinux is relevant for AIX is something you'll have to

"manylinux" per se, is not going to work for AIX - just as it could not work for FreeBSD, Windows, or macOS, etc. For a long long time - platform related code on aix was sys.platform.startswith("aix"), or something similar. FYI: starting with Python 3.8 - that will still work, but the return value is merely "aix" - to reflect that since mid 2007 IBM has "guaranteed" binary compatibility from old to new. So, the manylinux idea will work for AIX - as long as it is built on an older version of AIX. So, the parts of manylinux - where they have researched the standard libs that are binary compatible from old to new is provided by AIX (so not for OSS libs, as they may have, but should not have, dependencies). Of course, it gets complex - but looking for light, not darkness :)

...

decide. Ultimately it depends on how well the default rules in PEP 425

The PEP speaks about ``distutils.util.get_platform()`` and for AIX - this is broken. Noone (who loves AIX) has looked at it since 2007 (at least) when the rules changed, and perhaps since 1999 (or there abouts) when 64-bit support first appeared (in AIX 4.3). whether Python 2.7, or Python 3.9 ('master') the value is the same: root@x066:[/data/prj/python/python3-3.9]./python Python 3.9.0a0 (heads/pr_13772-dirty:66db707bb0, Aug 12 2019, 18:55:33) [C] on aix Type "help", "copyright", "credits" or "license" for more information.

...

...
...
import distutils import distutils.util distutils.util.get_platform() 'aix-6.1'

So, focusing on the "Platform Tag" - that needs to include more than what uname -a provides - the tag needs to provide the oslevel TL and SP (Technology Level, Service Pack) as well as whether it is 32 or 64 bit. 32-bit is still the most common "packaging" for AIX, but it needs to be specified - imho. I am guessing - that would be a PR for CPython - to get that into distutils. I have not read your other pointers (above), but I am going to guess that what follows would be addressed in setuptools. My question is - can you verify, or at least agree - that I need to take up getting the "Platform Tag" corrected in distutils - as a first step?

...

manage to ensure that binaries are interoperable, which in turn probably depends on how Python typically gets built on AIX, and how compatible dynamically linked system libraries are across machines.

Paul

On Tue, 13 Aug 2019 at 15:56, Michael wrote:

...
On 06/08/2019 09:15, Michael wrote: OK - been about a week - so, before I forget everything a list-ping...

Nick Coghlan

14 Aug 14 Aug

6:39 p.m.

On Wed, 14 Aug 2019 at 01:52, Michael wrote:

...

The PEP speaks about ``distutils.util.get_platform()`` and for AIX - this is broken. Noone (who loves AIX) has looked at it since 2007 (at least) when the rules changed, and perhaps since 1999 (or there abouts) when 64-bit support first appeared (in AIX 4.3).

whether Python 2.7, or Python 3.9 ('master') the value is the same: root@x066:[/data/prj/python/python3-3.9]./python Python 3.9.0a0 (heads/pr_13772-dirty:66db707bb0, Aug 12 2019, 18:55:33) [C] on aix Type "help", "copyright", "credits" or "license" for more information.

...
...
...
import distutils import distutils.util distutils.util.get_platform() 'aix-6.1'

So, focusing on the "Platform Tag" - that needs to include more than what uname -a provides - the tag needs to provide the oslevel TL and SP (Technology Level, Service Pack) as well as whether it is 32 or 64 bit. 32-bit is still the most common "packaging" for AIX, but it needs to be specified - imho.

I am guessing - that would be a PR for CPython - to get that into distutils.

Not really, since the platform tag generation needs to work on older Python versions. Instead, you'll want to define a new algorithm for deriving the platform tag on AIX that can be referenced from https://packaging.python.org/specifications/platform-compatibility-tags/ You shouldn't have the same challenges that manylinux did when it comes to determining a suitable build environment, though - IBM are already doing that work for AIX, just as MS do it for Windows, and Apple do it for Mac OS X. What you may need to constrain is the compiler and libc versions to specifically be the platform ones. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Michael

10:36 p.m.

On 14/08/2019 15:09, Nick Coghlan wrote:

...

...
I am guessing - that would be a PR for CPython - to get that into distutils. Not really, since the platform tag generation needs to work on older Python versions. OK. Instead, you'll want to define a new algorithm for deriving the platform tag on AIX that can be referenced from https://packaging.python.org/specifications/platform-compatibility-tags/

You shouldn't have the same challenges that manylinux did when it comes to determining a suitable build environment, though - IBM are already doing that work for AIX, just as MS do it for Windows, and Apple do it for Mac OS X.

I am probably missing something that is right in front of me. By "new algorithm" for generating a platform tag - I thought that would have been a PR to update the *.get_platform(). What should go into that seems fairly straight-forward to me. And it would/should reflect the "build environment" provided by IBM_AIX. But when I read https://packaging.python.org/specifications/platform-compatibility-tags/ I have the impression I am tasked with generating a new "tag", e.g., manyaix. I do not think a new pseudo-tag is needed - if the tag is generated properly. The danger comes from, e.g., a Python 3 executable built on AIX 7.1 and someone builds a module on AIX 7.2. Chances are (read I would expect) the module would not work on AIX 7.1. This is two different moments in time, lets assume two different 'packagers'. I can play around on my own - learning curve - using the tag as it is for now. I suspect that noone "AIX oriented" is using either eggs or wheels. They may be trying the format RPM (I tried with cffi and that format failed to complete) but others succeed. Just no wheels. And, is it a bug? if I use ./setup.py bdist -b aix-SOMETHING because in the dist directory the filenames are all using aix-6.1 where the platform tag is expected. All I see is that the directory aix-SOMETHING exists, and there is a message that aix-SOMETHING/dumb is removed. Regards, Michael

Nick Coghlan

17 Aug 17 Aug

5:36 p.m.

On Thu, 15 Aug 2019 at 03:03, Michael wrote:

...

On 14/08/2019 15:09, Nick Coghlan wrote:

I am guessing - that would be a PR for CPython - to get that into distutils.

Not really, since the platform tag generation needs to work on older Python versions.

OK.

Instead, you'll want to define a new algorithm for deriving the platform tag on AIX that can be referenced from https://packaging.python.org/specifications/platform-compatibility-tags/

You shouldn't have the same challenges that manylinux did when it comes to determining a suitable build environment, though - IBM are already doing that work for AIX, just as MS do it for Windows, and Apple do it for Mac OS X.

I am probably missing something that is right in front of me.

By "new algorithm" for generating a platform tag - I thought that would have been a PR to update the *.get_platform(). What should go into that seems fairly straight-forward to me. And it would/should reflect the "build environment" provided by IBM_AIX.

But when I read https://packaging.python.org/specifications/platform-compatibility-tags/ I have the impression I am tasked with generating a new "tag", e.g., manyaix.

I do not think a new pseudo-tag is needed - if the tag is generated properly. The danger comes from, e.g., a Python 3 executable built on AIX 7.1 and someone builds a module on AIX 7.2. Chances are (read I would expect) the module would not work on AIX 7.1. This is two different moments in time, lets assume two different 'packagers'.

It doesn't need to be a new pseudo-tag, it just needs to account for the fact that distutils.get_platform() in the standard library won't be changed to return a different answer on AIX when running older versions of Python. So, on the publishing side, at least setuptools and wheel will need updating to generate wheel archive properly for AIX. On the installation side, at least pip and the packaging project will need updating to generate the correct set of candidate platform tags. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Michael

18 Aug 18 Aug

5 p.m.

On 17/08/2019 14:06, Nick Coghlan wrote:

...

...
On 14/08/2019 15:09, Nick Coghlan wrote:

I am guessing - that would be a PR for CPython - to get that into distutils.

Not really, since the platform tag generation needs to work on older Python versions.

OK.

Instead, you'll want to define a new algorithm for deriving the platform tag on AIX that can be referenced from https://packaging.python.org/specifications/platform-compatibility-tags/

You shouldn't have the same challenges that manylinux did when it comes to determining a suitable build environment, though - IBM are already doing that work for AIX, just as MS do it for Windows, and Apple do it for Mac OS X.

I am probably missing something that is right in front of me.

By "new algorithm" for generating a platform tag - I thought that would have been a PR to update the *.get_platform(). What should go into that seems fairly straight-forward to me. And it would/should reflect the "build environment" provided by IBM_AIX.

But when I read https://packaging.python.org/specifications/platform-compatibility-tags/ I have the impression I am tasked with generating a new "tag", e.g., manyaix.

I do not think a new pseudo-tag is needed - if the tag is generated properly. The danger comes from, e.g., a Python 3 executable built on AIX 7.1 and someone builds a module on AIX 7.2. Chances are (read I would expect) the module would not work on AIX 7.1. This is two different moments in time, lets assume two different 'packagers'. It doesn't need to be a new pseudo-tag, it just needs to account for

On Thu, 15 Aug 2019 at 03:03, Michael wrote: the fact that distutils.get_platform() in the standard library won't be changed to return a different answer on AIX when running older versions of Python.

Would be 'nice' if old meant 3.6 and earlier as I would like to target 3.7 and later. PEP425 would not, imho, be contradicted - as it says "The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore _." My belief is that the current implementation for AIX is insufficient. e.g., it does not even include 32/64 bitness. I think I read your comment re: https://packaging.python.org/specifications/platform-compatibility-tags/ better, and I wonder how that would be done: a modification to the document (platform-compatibility-tags), or something else? The current information talks about the use of a psuedo-tag because the scheme defined in PEP425 - for linux - did not work for *linux* platform tags. I have done my research, and have a good idea of how and where to collect and store "build" related information, versus "run-time" information. For the record, I notice that sysconfig.get_platform() and distutils.util.get_platform() are 'real-time' oriented. So, I see no need to alter the nature of sysconfig.get_platform() or distutils.util.get_platform() - they can stay run-time. But some additional information needs to be provided. So, where is the best place to discuss what these system changes *should* be.

...

So, on the publishing side, at least setuptools and wheel will need updating to generate wheel archive properly for AIX.

Nods.

...

On the installation side, at least pip and the packaging project will need updating to generate the correct set of candidate platform tags.

I am looking at the links Paul suggested, and that gave me the food for thought needed to come up with an algorithm as you suggested. Trying to not repeat many communicative errors I made years back - getting something I saw as a bug-fix, but Python saw as a feature change - I am trying hard to get the discussion started properly. As to Python3.7 and Python3.8 - I am happy with a "maybe". I do not think it a correct approach if I were to make proposals for pip, setuptools and wheel projects that take - what Python gives is insufficient. And, I still need 'something' in Cpython, even if it is only a define in sysconfig.get_vars() so that there is a "build" related value. FYI GNU_HOST_TYPE is nearly complete for what is needed. But I slip towards details - so I'll stop for now. Reiterating - continue only here, or expand and include python-dev? Michael

...

Cheers, Nick.

1712

Age (days ago)

1729

Last active (days ago)

List overview

Download

11 comments

4 participants

participants (4)

Michael
Nick Coghlan
Paul Moore
Ronald Oussoren