Recently I have become more interested in looking at how to simplify
packaging for AIX.
Currently I more or less ignore anything pip and/or distutils could mean
for a python "user" (e.g., AIX sys admin and/or Python developer working
on AIX) as I just run "pip install XYZ" locally, find what constitutes
the equivalent of a bdist, and repackage those as 'installp' packages.
For someone with root access this is ok - as installp requires root (or
RBAC equivalent) to install.
As I would like to make installing modules more Python like - I asked
around and was pointed at piwheels (which looks very useful for my
aspiration) but also at PEP425 and the platform-tag.
There are several things I need to learn about dist-utils, and how
packaging tagging needs to be considered when, e.g., Python is built on
AIX 7.1 and the module is built on AIX 7.2 - there may be AIX ABI
differences that would prevent the module built on AIX 7.2 from working
properly on AIX 7.1 while the same module built on AIX 7.1 will run with
no issue on AIX 7.2. (A simplified explanation is that libc.a on AIX 7.2
knows how to work with applications linked against libc.a from earlier
versions of AIX, but earlier versions of AIX do not know how to deal
with libc.a from AIX 7.2.)
Taken to an extreme: Python(2|3) and modules built on AIX 5.3 TL7 run,
unaltered, on all levels of AIX including the latest AIX 7.2 - while my
expectation is that executable and modules built on AIX 7.2 TL5 (latest
level) might not run on AIX 7.2 TL4.
In short, "version", "release" are in themselves not enough - the TL
(technology level) should also be considered.
Additionally, what I also see "missing" is the
platform.architecture() value. By default, this is still 32bit on AIX
- but it is important - especially for pre-built eggs and/or wheels.
In the AIX world - the OS-level name scheme is usually: VR00-TL-SP
(Version, Release, TechnologyLevel, ServicePack). There is also a value
compareable to a builddate, but for distutils purposes has no value (I
can think of).
So, to my question: currently, for AIX get_host_platform returns
something such as: aix-6.1.
Considering above: what would you - as more expericenced with
multi-oslevel packaging and low levels are accepted by high-levels, but
"What should the AIX get_host_platform() string contain?"
At a minimum I forsee: return "%s-%s.%s-%s" % (osname, version,
But this does not address potential issues where the TL level within a
version.release has changed. (X.Y.TL5 built packages MIGHT work on
X.YTL4, but there is no reason to expect them to.
So, I would look to something that remains recognizable, but uses
e.g., oslevel -s returns a string such as: 6100-09-10-1731
Then using the equivalent of:
version, release, service, builddata = '6100-09-10-1731'.split('-')
return "%s-%s.%s.%s-%s" % (osname, version, release, service,
Note: no special authority is needed to run "oslevel -s", but it does
take time. So having a way, in the library to only have to actually call
for the value once - is a great improvement. I can imagine a way to do
it (store a static value somewhere, and when it is NULL aka not
initialized call the program, otherwise return the value) - but "WHERE"
- in distutils, or (ideally?) elsewhere in a more "central" library.
Starting a new thread - I think the first one has served it's purpose.
I use, successfully, pip build; pip install to build and install modules
for Python. The files "installed" by pip install (to .../site-packages)
I repackage as installp (AIX installp manager) packages.
This works - but - requires root access (to execute installp) and I do
not expect this will support anything like virtualenv.
Further, sometimes a .egg-info file is created, sometimes not. And when
it is created it does not always include an "AIX" tag.
As example, I show some data from the modules I "pip built and
installed" and then repackaged as installp:
root@x066:[/home/root]lslpp -L | grep python3 | grep -v adt | sort
aixtools.python3.rte 184.108.40.206 C F python python3
aixtools.python3.six.rte 220.127.116.11 C F tools six 26-Jul-2019
root@x066:[/home/root]find /opt/lib/python3.7 -name \*.egg\*
There are no .whl (except for pip and setuptools).
I am starting to guess that getting a wheel built requires more than
what most packages are doing.
Further, I am sure cffi should be including an AIX platform tag - as it
generates a .so file that depends on standard libraries from AIX (libc
I'll have read up on what is actually in the egg-info (using od -c I can
read the file).
Seems to be very close to what I find here
-rw-r--r-- 1 bin bin 4783 Jul 26 15:08 PKG-INFO
-rw-r--r-- 1 bin bin 11759 Jul 26 15:08 SOURCES.txt
-rw-r--r-- 1 bin bin 1 Jul 26 15:08
-rw-r--r-- 1 bin bin 144 Jul 26 15:08 native_libs.txt
-rw-r--r-- 1 bin bin 1 Jul 26 15:08 not-zip-safe
-rw-r--r-- 1 bin bin 381 Jul 26 15:08 requires.txt
-rw-r--r-- 1 bin bin 46 Jul 26 15:08 top_level.txt
In short, I am just a user of pip and setuptools - but I would like to
utilize them for packaging "binary" modules for AIX that do not require
installp (or the alternate package manager - rpm).
Some pointers into the documentation that is essential (I am starting at
https://setuptools.readthedocs.io/en/latest/, but seems like quite the
Thanks (for remembering what it was like when you first started!),
Well I think I found the culprit. I knew it must be something simple. And it was.
Our pydisutils.cfg had both an index-url and find_links, both with the same simple url.
When I removed the find_links, all the time sinks went away. What did take 3hr is now back to 24min. /facepalm
fwiw, this is just a python setup.py test, and uses tox virtualenv setuptools. I also thought that setuptools just used pip, but now I see it's actually easy-install, which I didn't realize was a different thing from pip.
Good luck on https://github.com/pypa/setuptools/issues/917
I have a python dev team that I support in our local jenkins/artifactory environment. Recently this team switched over from an old local pypi legacy repository to a current pypi remote/virtual of pypi.org. So we could get them updated libraries and such. Anyway, I am banging my head to explain the time taken for reading the simple index. This index is about 11MB, takes 1 second to download itself, but then it takes ~30 seconds for setuptools to scan this and find the package it's looking for. Multiple this by the ~170 libraries they have in requirements, and the build has gone from 30minutes to almost 3 hours. And 90% comes from reading the simple index for each dependency.
Here's a sample output with timestamps
00:57:00.036 Searching for tenacity==4.8.0
00:57:00.037 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/
00:57:27.650 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/tenacity/
00:57:27.881 Downloading https://repo1.local/artifactory/api/pypi/py-pypi-virt/packages/packages/fc/…
00:57:28.037 Best match: tenacity 4.8.0
00:57:28.037 Processing tenacity-4.8.0-py2.py3-none-any.whl
00:57:28.038 Installing tenacity-4.8.0-py2.py3-none-any.whl to /home/build/workspace/workflows-pr/workflows-extension/.eggs
00:57:28.069 writing requirements to /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg/EGG-INFO/requires.txt
00:57:28.114 Installed /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg
Note the time on the first Reading. I'm hoping there is some obvious simple solution to make this faster.
Any ideas or help would be greatly appreciated?
I was having a look at https://github.com/pypa/setuptools/issues/1473 but am unclear what is going wrong.
Running: (python3 setup.py sdist --formats zip) gives what i would expect to be the desired result.
[setup-tools-tes ~ 04:17] $ python3 setup.py sdist --formats zip
writing dependency_links to HelloWorld.egg-info/dependency_links.txt
writing top-level names to HelloWorld.egg-info/top_level.txt
reading manifest file 'HelloWorld.egg-info/SOURCES.txt'
writing manifest file 'HelloWorld.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md
warning: check: missing required meta-data: url
warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied
copying files to HelloWorld-0.1...
copying setup.py -> HelloWorld-0.1
copying HelloWorld.egg-info/PKG-INFO -> HelloWorld-0.1/HelloWorld.egg-info
copying HelloWorld.egg-info/SOURCES.txt -> HelloWorld-0.1/HelloWorld.egg-info
copying HelloWorld.egg-info/dependency_links.txt -> HelloWorld-0.1/HelloWorld.egg-info
copying HelloWorld.egg-info/top_level.txt -> HelloWorld-0.1/HelloWorld.egg-info
creating 'dist/HelloWorld-0.1.zip' and adding 'HelloWorld-0.1' to it
removing 'HelloWorld-0.1' (and everything under it)
[setup-tools-tes ~ 04:17] $ ls dist/