Mailman 3 - Distutils-SIG

It all works - wheels for AIX - up to a point
by Michael Aug. 15, 2019

Aug. 15, 2019

Spent some hours - experimenting... So - short story - it works. There remains a big chance for failure though as the current tag ignores differences in TL level, build_date, etc.. Also, a source for failure, is no recognition for 32-bit or 64-bit. However, focusing on the good news - the wheels install - without a compiler being called. Looking Forward!! AIX has a long history of letting binaries from the past be reused: See: https://www.ibm.com/support/knowledgecenter/ssw_aix_72/install/binary_compa… ++++++ The gritty details, for the curious/interested After having used devpi (client) to upload to a test devpi-server I pointed my pip at the devpi-server to download the packages. Note: using default pipy "source" I would get two .gz files. The initial upload is from an AIX 6.1 server, the download here is to an AIX 7.1 server. (testpi) devpi@x064:[/home/devpi/testpi]pip download -i http://x064:8441/aixtools/dev cffi Looking in indexes: http://x064:8441/aixtools/dev Collecting cffi Downloading http://x064:8441/aixtools/dev/%2Bf/504/244e6fc36188b/cffi-1.12.3.tar.gz (452kB) |################################| 460kB 28.7MB/s Saved ./cffi-1.12.3.tar.gz Collecting pycparser (from cffi) Downloading http://x064:8441/aixtools/dev/%2Bf/d9d/f2f3597ea75ab/pycparser-2.19-py2.py3… (111kB) |################################| 112kB 28.4MB/s Saved ./pycparser-2.19-py2.py3-none-any.whl Successfully downloaded cffi pycparser So, a wheel (for pycparser) and a .gz (for cffi). Phase 2: repeat the upload of cffi, but now from an AIX 7.1 server. Preview: (piserv) devpi@x064:[/home/devpi/testpi/cffi-1.12.3]devpi list pycparser http://x064:8441/aixtools/dev/+f/d9d/f2f3597ea75ab/pycparser-2.19-py2.py3-n… http://x064:8441/aixtools/dev/+f/2a8/90514e40f6d1c/pycparser-2.19.tar.gz (piserv) devpi@x064:[/home/devpi/testpi/cffi-1.12.3]devpi list cffi http://x064:8441/aixtools/dev/+f/53d/50f8d231592dc/cffi-1.12.3-cp37-cp37m-a… http://x064:8441/aixtools/dev/+f/504/244e6fc36188b/cffi-1.12.3.tar.gz http://x064:8441/aixtools/dev/+f/4d2/0d8ee28c40fe7/cffi-1.12.3-cp37-cp37m-a… (testpi) devpi@x064:[/home/devpi/xyz]pip download -i http://x064:8441/aixtools/dev cffi Looking in indexes: http://x064:8441/aixtools/dev Collecting cffi Downloading http://x064:8441/aixtools/dev/%2Bf/4d2/0d8ee28c40fe7/cffi-1.12.3-cp37-cp37m… (197kB) |################################| 204kB 20.7MB/s Saved ./cffi-1.12.3-cp37-cp37m-aix_7_1.whl Collecting pycparser (from cffi) Downloading http://x064:8441/aixtools/dev/%2Bf/d9d/f2f3597ea75ab/pycparser-2.19-py2.py3… (111kB) |################################| 112kB 23.3MB/s Saved ./pycparser-2.19-py2.py3-none-any.whl Successfully downloaded cffi pycparser Regards, Michael

1 0

setuptools (was distutils...) and AIX - getting to PEP427 or PEP491
by Michael Aug. 14, 2019

Aug. 14, 2019

Starting a new thread - I think the first one has served it's purpose. I use, successfully, pip build; pip install to build and install modules for Python. The files "installed" by pip install (to .../site-packages) I repackage as installp (AIX installp manager) packages. This works - but - requires root access (to execute installp) and I do not expect this will support anything like virtualenv. Further, sometimes a .egg-info file is created, sometimes not. And when it is created it does not always include an "AIX" tag. As example, I show some data from the modules I "pip built and installed" and then repackaged as installp: root@x066:[/home/root]lslpp -L | grep python3 | grep -v adt | sort aixtools.python3.asn1crypto.rte aixtools.python3.bcrypt.rte aixtools.python3.cffi.rte aixtools.python3.cryptography.rte aixtools.python3.paramiko.rte aixtools.python3.pycparser.rte aixtools.python3.pynacl.rte aixtools.python3.rte 3.7.4.0 C F python python3 13-Aug-2019 aixtools.python3.six.rte 1.12.0.0 C F tools six 26-Jul-2019 root@x066:[/home/root]find /opt/lib/python3.7 -name \*.egg\* /opt/lib/python3.7/site-packages/cffi-1.12.3-py3.7.egg-info /opt/lib/python3.7/site-packages/cryptography-2.7-py3.7-aix-6.1.egg /opt/lib/python3.7/site-packages/pycparser-2.19-py3.7.egg-info There are no .whl (except for pip and setuptools). I am starting to guess that getting a wheel built requires more than what most packages are doing. Further, I am sure cffi should be including an AIX platform tag - as it generates a .so file that depends on standard libraries from AIX (libc and libpthreads). I'll have read up on what is actually in the egg-info (using od -c I can read the file). Seems to be very close to what I find here root@x066:[/opt/lib/python3.7/site-packages/cryptography-2.7-py3.7-aix-6.1.egg/EGG-INFO]ls -l total 80 -rw-r--r-- 1 bin bin 4783 Jul 26 15:08 PKG-INFO -rw-r--r-- 1 bin bin 11759 Jul 26 15:08 SOURCES.txt -rw-r--r-- 1 bin bin 1 Jul 26 15:08 dependency_links.txt -rw-r--r-- 1 bin bin 144 Jul 26 15:08 native_libs.txt -rw-r--r-- 1 bin bin 1 Jul 26 15:08 not-zip-safe -rw-r--r-- 1 bin bin 381 Jul 26 15:08 requires.txt -rw-r--r-- 1 bin bin 46 Jul 26 15:08 top_level.txt ++++++ In short, I am just a user of pip and setuptools - but I would like to utilize them for packaging "binary" modules for AIX that do not require installp (or the alternate package manager - rpm). Some pointers into the documentation that is essential (I am starting at https://setuptools.readthedocs.io/en/latest/, but seems like quite the mountain!). Thanks (for remembering what it was like when you first started!), Michael

3 2

PyPI security work: multifactor auth progress & help needed
by Sumana Harihareswara Aug. 6, 2019

Aug. 6, 2019

Work has started on the Open Technology Fund-supported project to improve Warehouse security, accessibility, and internationalization. More details in today's progress report: https://discuss.python.org/t/pypi-security-work-multifactor-auth-progress-h… best, Sumana Harihareswara Warehouse project manager Changeset Consulting

2 6

Re: setuptools reading simple index takes 30sec
by Cralle, Chris Aug. 2, 2019

Aug. 2, 2019

Well I think I found the culprit. I knew it must be something simple. And it was. Our pydisutils.cfg had both an index-url and find_links, both with the same simple url. When I removed the find_links, all the time sinks went away. What did take 3hr is now back to 24min. /facepalm fwiw, this is just a python setup.py test, and uses tox virtualenv setuptools. I also thought that setuptools just used pip, but now I see it's actually easy-install, which I didn't realize was a different thing from pip. Good luck on https://github.com/pypa/setuptools/issues/917

1 0

setuptools reading simple index takes 30sec
by chris.cralle＠rsa.com Aug. 2, 2019

Aug. 2, 2019

Hello, I have a python dev team that I support in our local jenkins/artifactory environment. Recently this team switched over from an old local pypi legacy repository to a current pypi remote/virtual of pypi.org. So we could get them updated libraries and such. Anyway, I am banging my head to explain the time taken for reading the simple index. This index is about 11MB, takes 1 second to download itself, but then it takes ~30 seconds for setuptools to scan this and find the package it's looking for. Multiple this by the ~170 libraries they have in requirements, and the build has gone from 30minutes to almost 3 hours. And 90% comes from reading the simple index for each dependency. Here's a sample output with timestamps 00:57:00.036 Searching for tenacity==4.8.0 00:57:00.037 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/ 00:57:27.650 Reading https://repo1.local/artifactory/api/pypi/py-pypi-virt/simple/tenacity/ 00:57:27.881 Downloading https://repo1.local/artifactory/api/pypi/py-pypi-virt/packages/packages/fc/… 00:57:28.037 Best match: tenacity 4.8.0 00:57:28.037 Processing tenacity-4.8.0-py2.py3-none-any.whl 00:57:28.038 Installing tenacity-4.8.0-py2.py3-none-any.whl to /home/build/workspace/workflows-pr/workflows-extension/.eggs 00:57:28.069 writing requirements to /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg/EGG-INFO/requires.txt 00:57:28.114 Installed /home/build/workspace/workflows-pr/workflows-extension/.eggs/tenacity-4.8.0-py2.7.egg Note the time on the first Reading. I'm hoping there is some obvious simple solution to make this faster. Any ideas or help would be greatly appreciated? Thanks

3 2

Issue 1473 (build_sdist expects only .tar.gz)
by A_Rog Aug. 1, 2019

Aug. 1, 2019

I was having a look at https://github.com/pypa/setuptools/issues/1473 but am unclear what is going wrong. Running: (python3 setup.py sdist --formats zip) gives what i would expect to be the desired result. [setup-tools-tes ~ 04:17] $ python3 setup.py sdist --formats zip running sdist running egg_info writing HelloWorld.egg-info/PKG-INFO writing dependency_links to HelloWorld.egg-info/dependency_links.txt writing top-level names to HelloWorld.egg-info/top_level.txt reading manifest file 'HelloWorld.egg-info/SOURCES.txt' writing manifest file 'HelloWorld.egg-info/SOURCES.txt' warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md running check warning: check: missing required meta-data: url warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied creating HelloWorld-0.1 creating HelloWorld-0.1/HelloWorld.egg-info copying files to HelloWorld-0.1... copying setup.py -> HelloWorld-0.1 copying HelloWorld.egg-info/PKG-INFO -> HelloWorld-0.1/HelloWorld.egg-info copying HelloWorld.egg-info/SOURCES.txt -> HelloWorld-0.1/HelloWorld.egg-info copying HelloWorld.egg-info/dependency_links.txt -> HelloWorld-0.1/HelloWorld.egg-info copying HelloWorld.egg-info/top_level.txt -> HelloWorld-0.1/HelloWorld.egg-info Writing HelloWorld-0.1/setup.cfg creating 'dist/HelloWorld-0.1.zip' and adding 'HelloWorld-0.1' to it adding 'HelloWorld-0.1' adding 'HelloWorld-0.1/HelloWorld.egg-info' adding 'HelloWorld-0.1/setup.py' adding 'HelloWorld-0.1/PKG-INFO' adding 'HelloWorld-0.1/setup.cfg' adding 'HelloWorld-0.1/HelloWorld.egg-info/SOURCES.txt' adding 'HelloWorld-0.1/HelloWorld.egg-info/top_level.txt' adding 'HelloWorld-0.1/HelloWorld.egg-info/dependency_links.txt' adding 'HelloWorld-0.1/HelloWorld.egg-info/PKG-INFO' removing 'HelloWorld-0.1' (and everything under it) [setup-tools-tes ~ 04:17] $ ls dist/ HelloWorld-0.1.zip

1 0

BETA: PyPI now supports API Tokens for Release uploads
by Ernest W. Durbin III July 25, 2019

July 25, 2019

PyPI and Test PyPI now support the creation of API Tokens for use when uploading projects to PyPI, thanks to work funded by the Open Technology Fund. These tokens are created by default with the same upload permissions as the User creating them, but can also be scoped to specific projects that User has upload privileges for. This is the first step in enforcing that Users with Two-Factor Authentication enabled must use an API Token when uploading to PyPI, rather than their password. After the Beta we’ll announce the general availability of these features and timeline for enforcement of API Tokens for Two-Factor Authentication enabled accounts. Read more on how you can help to test this feature at: https://discuss.python.org/t/pypi-security-work-multifactor-auth-progress-h… -Ernest W. Durbin III Director of Infrastructure Python Software Foundation

2 1

pip 19.2 is now available!
by Pradyun Gedam July 23, 2019

July 23, 2019

Hello everyone! On behalf of the PyPA, I am pleased to announce that pip 19.2 has just been released. The highlights of this release are: - Python 3.4 support has been dropped - Support for "yanked" files on package indexes (PEP 592) - keyring can now be used for managing credentials - A new `pip debug` command, for helping debug installations - Many bug fixes and lots of minor improvements We've also documented how Python 2 support in pip, would be maintained going forward. [1] To install pip 19.2, you can use get-pip (as described in [2]) or run: python -m pip install --upgrade pip Note that if you are using a version of pip supplied by your distribution vendor, vendor-supplied upgrades will be available in due course. The pip development team is extremely grateful to everyone in the community for their contributions. Thanks to everyone who put so much effort into the new release. Many of the contributions came from community members, whether in the form of code, participation in design discussions and/or bug reports. [1]: https://pip.pypa.io/en/latest/development/release-process/#python-2-support [2]: https://pip.pypa.io/en/latest/installing

1 0

Seeking a new maintainer for packaging.python.org and Twine.
by Thea Flowers July 22, 2019

July 22, 2019

Hi friends! I'm stepping away from several things in the Python community. I've served as the maintainer of packaging.python.org and Twine for a while, but it's time for me to move on. I'm looking for someone to take on primary responsibility for packaging.python.org and another person to help share the load for Twine. If you or someone you know is interested, please reply to this thread and let me know. -Thea Flowers magicalgirl(a)google.com

9 10

WIll custom setup.py with (set library-prefix have post install actions) continue to be supported ?
by Stuart Axon June 23, 2019

June 23, 2019

I'm making a python package for something python Gedit/Pluma/Xed plugin. Like other lots of other packages that use Gtk, I use glib-compile-schemas [1] at install time. I do this by using a custom setup.py- similar to this file this from flux: https://github.com/xflux-gui/fluxgui/blob/master/setup.py#L81(other app, like Meld to something similar) It looks like there are a lot of changes happening to setuptools, will this continue to work or will it break one day ? This isn't the only file I'd like to generate at install time, but might be the hardest to work around if I can't. The other custom action I'm going looking at is setting `--install-lib` to the editors plugin directory (this varies depending on: Operating system and is the user root ?) I'm using use pip for this, as the install story for most python plugins for these editors is nonexistant - people have to copy the files to some place manually, and instructions usually miss out Windows and OSX. [1] glib-compile-schemas takes an XML file and generates a binary GVariant file.

5 4