Mailman 3 August 2019 - Distutils-SIG

distutils.util.get_host_platform() and AIX
by Michael Aug. 18, 2019

Aug. 18, 2019

Recently I have become more interested in looking at how to simplify packaging for AIX. Currently I more or less ignore anything pip and/or distutils could mean for a python "user" (e.g., AIX sys admin and/or Python developer working on AIX) as I just run "pip install XYZ" locally, find what constitutes the equivalent of a bdist, and repackage those as 'installp' packages. For someone with root access this is ok - as installp requires root (or RBAC equivalent) to install. As I would like to make installing modules more Python like - I asked around and was pointed at piwheels (which looks very useful for my aspiration) but also at PEP425 and the platform-tag. There are several things I need to learn about dist-utils, and how packaging tagging needs to be considered when, e.g., Python is built on AIX 7.1 and the module is built on AIX 7.2 - there may be AIX ABI differences that would prevent the module built on AIX 7.2 from working properly on AIX 7.1 while the same module built on AIX 7.1 will run with no issue on AIX 7.2. (A simplified explanation is that libc.a on AIX 7.2 knows how to work with applications linked against libc.a from earlier versions of AIX, but earlier versions of AIX do not know how to deal with libc.a from AIX 7.2.) Taken to an extreme: Python(2|3) and modules built on AIX 5.3 TL7 run, unaltered, on all levels of AIX including the latest AIX 7.2 - while my expectation is that executable and modules built on AIX 7.2 TL5 (latest level) might not run on AIX 7.2 TL4. In short, "version", "release" are in themselves not enough - the TL (technology level) should also be considered. Additionally, what I also see "missing" is the platform.architecture()[0] value. By default, this is still 32bit on AIX - but it is important - especially for pre-built eggs and/or wheels. In the AIX world - the OS-level name scheme is usually: VR00-TL-SP (Version, Release, TechnologyLevel, ServicePack). There is also a value compareable to a builddate, but for distutils purposes has no value (I can think of). So, to my question: currently, for AIX get_host_platform returns something such as: aix-6.1. Considering above: what would you - as more expericenced with multi-oslevel packaging and low levels are accepted by high-levels, but not v.v. "What should the AIX get_host_platform() string contain?" At a minimum I forsee: return "%s-%s.%s-%s" % (osname, version, release, platform.architecture()[0]) But this does not address potential issues where the TL level within a version.release has changed. (X.Y.TL5 built packages MIGHT work on X.YTL4, but there is no reason to expect them to. So, I would look to something that remains recognizable, but uses different 'punctuation' e.g., oslevel -s returns a string such as: 6100-09-10-1731 Then using the equivalent of: version, release, service, builddata = '6100-09-10-1731'.split('-') return "%s-%s.%s.%s-%s" % (osname, version, release, service, platform.architecture()[0]) Note: no special authority is needed to run "oslevel -s", but it does take time. So having a way, in the library to only have to actually call for the value once - is a great improvement. I can imagine a way to do it (store a static value somewhere, and when it is NULL aka not initialized call the program, otherwise return the value) - but "WHERE" - in distutils, or (ideally?) elsewhere in a more "central" library. Sincerely, Michael

4 11

It all works - wheels for AIX - up to a point
by Michael Aug. 15, 2019

Aug. 15, 2019

Spent some hours - experimenting... So - short story - it works. There remains a big chance for failure though as the current tag ignores differences in TL level, build_date, etc.. Also, a source for failure, is no recognition for 32-bit or 64-bit. However, focusing on the good news - the wheels install - without a compiler being called. Looking Forward!! AIX has a long history of letting binaries from the past be reused: See: https://www.ibm.com/support/knowledgecenter/ssw_aix_72/install/binary_compa… ++++++ The gritty details, for the curious/interested After having used devpi (client) to upload to a test devpi-server I pointed my pip at the devpi-server to download the packages. Note: using default pipy "source" I would get two .gz files. The initial upload is from an AIX 6.1 server, the download here is to an AIX 7.1 server. (testpi) devpi@x064:[/home/devpi/testpi]pip download -i http://x064:8441/aixtools/dev cffi Looking in indexes: http://x064:8441/aixtools/dev Collecting cffi Downloading http://x064:8441/aixtools/dev/%2Bf/504/244e6fc36188b/cffi-1.12.3.tar.gz (452kB) |################################| 460kB 28.7MB/s Saved ./cffi-1.12.3.tar.gz Collecting pycparser (from cffi) Downloading http://x064:8441/aixtools/dev/%2Bf/d9d/f2f3597ea75ab/pycparser-2.19-py2.py3… (111kB) |################################| 112kB 28.4MB/s Saved ./pycparser-2.19-py2.py3-none-any.whl Successfully downloaded cffi pycparser So, a wheel (for pycparser) and a .gz (for cffi). Phase 2: repeat the upload of cffi, but now from an AIX 7.1 server. Preview: (piserv) devpi@x064:[/home/devpi/testpi/cffi-1.12.3]devpi list pycparser http://x064:8441/aixtools/dev/+f/d9d/f2f3597ea75ab/pycparser-2.19-py2.py3-n… http://x064:8441/aixtools/dev/+f/2a8/90514e40f6d1c/pycparser-2.19.tar.gz (piserv) devpi@x064:[/home/devpi/testpi/cffi-1.12.3]devpi list cffi http://x064:8441/aixtools/dev/+f/53d/50f8d231592dc/cffi-1.12.3-cp37-cp37m-a… http://x064:8441/aixtools/dev/+f/504/244e6fc36188b/cffi-1.12.3.tar.gz http://x064:8441/aixtools/dev/+f/4d2/0d8ee28c40fe7/cffi-1.12.3-cp37-cp37m-a… (testpi) devpi@x064:[/home/devpi/xyz]pip download -i http://x064:8441/aixtools/dev cffi Looking in indexes: http://x064:8441/aixtools/dev Collecting cffi Downloading http://x064:8441/aixtools/dev/%2Bf/4d2/0d8ee28c40fe7/cffi-1.12.3-cp37-cp37m… (197kB) |################################| 204kB 20.7MB/s Saved ./cffi-1.12.3-cp37-cp37m-aix_7_1.whl Collecting pycparser (from cffi) Downloading http://x064:8441/aixtools/dev/%2Bf/d9d/f2f3597ea75ab/pycparser-2.19-py2.py3… (111kB) |################################| 112kB 23.3MB/s Saved ./pycparser-2.19-py2.py3-none-any.whl Successfully downloaded cffi pycparser Regards, Michael

1 0

setuptools (was distutils...) and AIX - getting to PEP427 or PEP491
by Michael Aug. 14, 2019

Aug. 14, 2019

Starting a new thread - I think the first one has served it's purpose. I use, successfully, pip build; pip install to build and install modules for Python. The files "installed" by pip install (to .../site-packages) I repackage as installp (AIX installp manager) packages. This works - but - requires root access (to execute installp) and I do not expect this will support anything like virtualenv. Further, sometimes a .egg-info file is created, sometimes not. And when it is created it does not always include an "AIX" tag. As example, I show some data from the modules I "pip built and installed" and then repackaged as installp: root@x066:[/home/root]lslpp -L | grep python3 | grep -v adt | sort aixtools.python3.asn1crypto.rte aixtools.python3.bcrypt.rte aixtools.python3.cffi.rte aixtools.python3.cryptography.rte aixtools.python3.paramiko.rte aixtools.python3.pycparser.rte aixtools.python3.pynacl.rte aixtools.python3.rte 3.7.4.0 C F python python3 13-Aug-2019 aixtools.python3.six.rte 1.12.0.0 C F tools six 26-Jul-2019 root@x066:[/home/root]find /opt/lib/python3.7 -name \*.egg\* /opt/lib/python3.7/site-packages/cffi-1.12.3-py3.7.egg-info /opt/lib/python3.7/site-packages/cryptography-2.7-py3.7-aix-6.1.egg /opt/lib/python3.7/site-packages/pycparser-2.19-py3.7.egg-info There are no .whl (except for pip and setuptools). I am starting to guess that getting a wheel built requires more than what most packages are doing. Further, I am sure cffi should be including an AIX platform tag - as it generates a .so file that depends on standard libraries from AIX (libc and libpthreads). I'll have read up on what is actually in the egg-info (using od -c I can read the file). Seems to be very close to what I find here root@x066:[/opt/lib/python3.7/site-packages/cryptography-2.7-py3.7-aix-6.1.egg/EGG-INFO]ls -l total 80 -rw-r--r-- 1 bin bin 4783 Jul 26 15:08 PKG-INFO -rw-r--r-- 1 bin bin 11759 Jul 26 15:08 SOURCES.txt -rw-r--r-- 1 bin bin 1 Jul 26 15:08 dependency_links.txt -rw-r--r-- 1 bin bin 144 Jul 26 15:08 native_libs.txt -rw-r--r-- 1 bin bin 1 Jul 26 15:08 not-zip-safe -rw-r--r-- 1 bin bin 381 Jul 26 15:08 requires.txt -rw-r--r-- 1 bin bin 46 Jul 26 15:08 top_level.txt ++++++ In short, I am just a user of pip and setuptools - but I would like to utilize them for packaging "binary" modules for AIX that do not require installp (or the alternate package manager - rpm). Some pointers into the documentation that is essential (I am starting at https://setuptools.readthedocs.io/en/latest/, but seems like quite the mountain!). Thanks (for remembering what it was like when you first started!), Michael

3 2

PyPI security work: multifactor auth progress & help needed
by Sumana Harihareswara Aug. 6, 2019

Aug. 6, 2019

Work has started on the Open Technology Fund-supported project to improve Warehouse security, accessibility, and internationalization. More details in today's progress report: https://discuss.python.org/t/pypi-security-work-multifactor-auth-progress-h… best, Sumana Harihareswara Warehouse project manager Changeset Consulting

2 6

Re: setuptools reading simple index takes 30sec
by Cralle, Chris Aug. 2, 2019

Aug. 2, 2019

Well I think I found the culprit. I knew it must be something simple. And it was. Our pydisutils.cfg had both an index-url and find_links, both with the same simple url. When I removed the find_links, all the time sinks went away. What did take 3hr is now back to 24min. /facepalm fwiw, this is just a python setup.py test, and uses tox virtualenv setuptools. I also thought that setuptools just used pip, but now I see it's actually easy-install, which I didn't realize was a different thing from pip. Good luck on https://github.com/pypa/setuptools/issues/917

1 0

setuptools reading simple index takes 30sec
by chris.cralle＠rsa.com Aug. 2, 2019

Aug. 2, 2019

Hello, I have a python dev team that I support in our local jenkins/artifactory environment. Recently this team switched over from an old local pypi legacy repository to a current pypi remote/virtual of pypi.org. So we could get them updated libraries and such. Anyway, I am banging my head to explain the time taken for reading the simple index. This index is about 11MB, takes 1 second to download itself, but then it takes ~30 seconds for setuptools to scan this and find the package it's looking for. Multiple this by the ~170 libraries they have in requirements, and the build has gone from 30minutes to almost 3 hours. And 90% comes from reading the simple index for each dependency. Here's a sample output with timestamps 00:57:00.036 Searching for tenacity==4.8.0 00:57:00.037 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/ 00:57:27.650 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/tenacity/ 00:57:27.881 Downloading https://repo1.local/artifactory/api/pypi/py-pypi-virt/packages/packages/fc/… 00:57:28.037 Best match: tenacity 4.8.0 00:57:28.037 Processing tenacity-4.8.0-py2.py3-none-any.whl 00:57:28.038 Installing tenacity-4.8.0-py2.py3-none-any.whl to /home/build/workspace/workflows-pr/workflows-extension/.eggs 00:57:28.069 writing requirements to /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg/EGG-INFO/requires.txt 00:57:28.114 Installed /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg Note the time on the first Reading. I'm hoping there is some obvious simple solution to make this faster. Any ideas or help would be greatly appreciated? Thanks

3 2

Issue 1473 (build_sdist expects only .tar.gz)
by A_Rog Aug. 1, 2019

Aug. 1, 2019

I was having a look at https://github.com/pypa/setuptools/issues/1473 but am unclear what is going wrong. Running: (python3 setup.py sdist --formats zip) gives what i would expect to be the desired result. [setup-tools-tes ~ 04:17] $ python3 setup.py sdist --formats zip running sdist running egg_info writing HelloWorld.egg-info/PKG-INFO writing dependency_links to HelloWorld.egg-info/dependency_links.txt writing top-level names to HelloWorld.egg-info/top_level.txt reading manifest file 'HelloWorld.egg-info/SOURCES.txt' writing manifest file 'HelloWorld.egg-info/SOURCES.txt' warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md running check warning: check: missing required meta-data: url warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied creating HelloWorld-0.1 creating HelloWorld-0.1/HelloWorld.egg-info copying files to HelloWorld-0.1... copying setup.py -> HelloWorld-0.1 copying HelloWorld.egg-info/PKG-INFO -> HelloWorld-0.1/HelloWorld.egg-info copying HelloWorld.egg-info/SOURCES.txt -> HelloWorld-0.1/HelloWorld.egg-info copying HelloWorld.egg-info/dependency_links.txt -> HelloWorld-0.1/HelloWorld.egg-info copying HelloWorld.egg-info/top_level.txt -> HelloWorld-0.1/HelloWorld.egg-info Writing HelloWorld-0.1/setup.cfg creating 'dist/HelloWorld-0.1.zip' and adding 'HelloWorld-0.1' to it adding 'HelloWorld-0.1' adding 'HelloWorld-0.1/HelloWorld.egg-info' adding 'HelloWorld-0.1/setup.py' adding 'HelloWorld-0.1/PKG-INFO' adding 'HelloWorld-0.1/setup.cfg' adding 'HelloWorld-0.1/HelloWorld.egg-info/SOURCES.txt' adding 'HelloWorld-0.1/HelloWorld.egg-info/top_level.txt' adding 'HelloWorld-0.1/HelloWorld.egg-info/dependency_links.txt' adding 'HelloWorld-0.1/HelloWorld.egg-info/PKG-INFO' removing 'HelloWorld-0.1' (and everything under it) [setup-tools-tes ~ 04:17] $ ls dist/ HelloWorld-0.1.zip

1 0