Mailman 3 - Distutils-SIG

travis problem
by Robin Becker 17 Sep '19

17 Sep '19

Channelling the experts for advice on travis. After being absent for a couple of weeks my travis builds have started failing The problem is seems to be related to this message in the failing builds > 3.5 is not installed; attempting download this is later followed by > Downloading archive: https://storage.googleapis.com/travis-ci-language-archives/python/binaries/… > 0.48s$ curl -sSf -o python-3.5.tar.bz2 ${archive_url} > 13.65s$ sudo tar xjf python-3.5.tar.bz2 --directory / my base python in .travis.yml is > python: 3.5 and in the matrix I have this > matrix: > exclude: > # Exclude the default Python 3.5 build > - python: 3.5 > include: > - os: linux > env: > - UNICODE_WIDTH=16 > - MB_PYTHON_VERSION=2.7 > - os: linux > env: ........ ............... I see no pattern in which are failing and which succeed -- Robin Becker

2 3

Python 3.7.4 MSI
by Renegad3 Kay 09 Sep '19

09 Sep '19

Greetings! my organization uses Python across it's departments but the recent versions of Python do NOT have an MSI download. We use SCCM for deployment of software and because the downloads are all .exe based, the program is no longer in compliance with my organization's security policies. I've looked and I've looked but I cannot find an MSI for this program. IT Guy.

4 3

localization, accessibility, & security progress on PyPI
by Sumana Harihareswara 08 Sep '19

08 Sep '19

I've just posted a few progress reports on Discourse about the last month of Open Tech Fund-supported progress on PyPI's localization, accessibility, & security features. https://discuss.python.org/t/pypi-localization-accessibility-progress/2284 https://discuss.python.org/t/pypi-security-work-multifactor-auth-progress-h… We've shifted our focus from security work to accessibility & internationalization work. We're aiming to wrap it up by September 30th. -- Sumana Harihareswara PyPI project manager Changeset Consulting https://changeset.nyc

1 0

PyPI & cryptographic signing and malware detection - seeking comment
by Sumana Harihareswara 03 Sep '19

03 Sep '19

Python Software Foundation has published a Request for Information seeking software developers to add these features to Warehouse (PyPI): * Verifiable cryptographic signing of artifacts (PEP 458/TUF or simiilar) * Technical infrastructure and methods for automated detection of malicious package uploads More info: https://github.com/python/request-for/blob/master/2019-Q4-PyPI/RFI.md We'd like for potential contractors & other experts to keep discussion at the Discourse forum https://discuss.python.org/c/python-software-foundation/pypi-q4-rfi , especially on these questions: • What methods should we implement to detect malicious content? https://discuss.python.org/t/what-methods-should-we-implement-to-detect-mal… and * PEPs 458 and 480 offer different levels of security; which (if either) should we implement? Which one has more appropriate operational efficacy? Should we use TUF (The Update Framework) or another approach? https://discuss.python.org/t/which-cryptographic-signing-approach/2241 and more generally: * What should community acceptance criteria be? * How feasible is it to implement this on PyPI? * What features do PyPI administrators need to make use of these features in the future? * What work would the developer need to do to make these features more maintainable by future Warehouse maintainers? -- Sumana Harihareswara PyPI project manager Changeset Consulting https://changeset.nyc

1 1

Linux binary wheels?
by Dan Stromberg 29 Aug '19

29 Aug '19

Hi folks. I have a pair of ideas about Linux binary wheels, which are currently (I heard) unsupported. It seems like it should be possible to support Linux binary wheels using one or both of these technologies: * https://build.opensuse.org/ is a service that builds packages for a variety of Linuxes * Docker could be used to automate the building of wheels for a handful of Linuxes with minimal dependencies. It seems like if you get Debian/Ubuntu/Mint, Fedora/CentOS, openSUSE and perhaps one or two others, that would cover almost all Linuxes and Linux users. I'm up to my hears in commitments already, but I sincerely someone will grab onto one or both of these possibilities and run with them. Thanks for reading.

10 15

timeline for new pip resolver rollout - feedback?
by Sumana Harihareswara 29 Aug '19

29 Aug '19

What timeline are we thinking is realistic for rolling out the new pip resolver? (latest update on resolver work: https://pradyunsg.me/blog/2019/08/06/pip-update-2/ ) I'm re-upping this question which I originally asked on a GitHub issue about the rollout: https://github.com/pypa/pip/issues/6536#issuecomment-521696430 and would prefer to corral answers there. This depends a lot on Pradyun's health and free time, and code review availability from other pip maintainers, and whether we get some grants we're applying for, but I think the sequence is something like: 1) build logic refactor: in progress, done sometime December-February 2) UX research and design, test infrastructure building, talking to downstreams and users about config flags and transition schedules: we need funding for this; earliest start is probably December, will take 2-3 months 3) introduce the abstractions defined in resolvelib/zazo while doing alpha testing: will take a few months, so, conservatively estimating, May 2020? 4) adopting better dependency resolution and do beta testing: ? Is this right? What am I missing? I ask because some of the info-gathering work is stuff a project manager and/or UX researcher should do, in my opinion, and because some progress on the increase in metadata strictness https://github.com/pypa/packaging-problems/issues/264 and other issues might help with concerns people have brought up here. -- Sumana Harihareswara PyPI project manager, PyPA member & coordinator, and person who seems to write a lot of grant applications Changeset Consulting https://changeset.nyc

2 1

pip and PEP 517 Backends
by Phil Thompson 25 Aug '19

25 Aug '19

I have a PEP 517 compatible backend which works with pip to install from an sdist (via an internal wheel). However there are a couple of issues... pip swallows all output from the backend. Is there anyway for the user to see the output (builds can take several minutes)? I would like to pass options from the pip command line to the backend but neither --global-option or --install-option have any effect (the config_settings argument is always None). How do I achieve this? Thanks, Phil

3 4

Adding support for adequately tagging AIX (pep425) to support distributed wheels
by Michael 19 Aug '19

19 Aug '19

Goal - using wheels rather than RPM and/or installp formats for distributing binary modules Why bother? One reason (there are likely more) - using wheels means packages/modules can be loaded in a virtualenv rather than require they are first loaded in the system environment using installp/rpm/yum (with root authority). This alone has been reason enough for me to do the research. Introduction The use of pip and wheels is commonplace in the worlds of Linux, macOS and Windows. Not so for AIX. Not because it couldn't be commonplace. The current situation for AIX is comparable to the initial issue Linux was facing when PEP513 was written: "Currently, distribution of binary Python extensions for Windows and OS X is straightforward. ... For Linux, the situation is much more delicate. ... Build tools using PEP 425 platform tags [3] do not track information about the particular Linux distribution or installed system libraries, and instead assign all wheels the too-vague linux_i686 or linux_x86_64 tags. Because of this ambiguity, ..." The root cause for the *ambiguity* that Linux systems had is not the ambiguity that AIX faces. AIX has provided a consistent way to "tag" it's runtime environment since at least 2007 (AIX 5.3 TL7). Since that time IBM AIX has also *guaranteed* binary compatibility for migration of applications from old to new OS levels. I would like to see these tags added - at a minimum that they can be retrieved by something such as sysconfig.get_var('AIC_BLD_TAG'). It would be "nice" to see sysconfig.get_platform() updated to include these values from the running system. Further, while pip related tools can add the "bitness" to the platform tags I would like to see something added to the AIX get_platform() tag (b32, ppc, aix32), (b64, ppc64, aix64) for 32 and 64 bit operations, respectively - as that is a "running" environment attribute. Open to other ideas on what the bitness tag should be. IMHO - anything is better than nothing. Maybe this could be considered a bug rather than as a new feature. Thank you for your feedback. Michael

2 1

Re: distutils.util.get_host_platform() and AIX
by Pradyun Gedam 19 Aug '19

19 Aug '19

Resending since I messed up the reply. On Sun, 18 Aug 2019 at 8:29 PM, Michael <aixtools(a)felt.demon.nl> wrote: > Would be 'nice' if old meant 3.6 and earlier as I would like to target 3.7 > and later. > I understand we're talking about generating the PEP 425 tags here. If not, please ignore this next paragraph. The "algorithm" essentially needs to be a check for "is this an AIX python" and also generate/provide a signal of what the system/binary compatibility looks like. This algorithm would executed on a target Python interpreter (which can be anything from Python 2.7, 3.3, 3.7 or a hypothetical 3.10), to check if that interpreter compatible with whatever tag you plan to generate. Reiterating - continue only here, or expand and include python-dev? > Continue here. python-dev has, practically, delegated Packaging discussions to this list. (opinion) If you're okay with changing communication channel, I have a strong preference for using discuss.python.org/c/packaging instead. Anyway, most of the packaging-related discussions have been happening there lately. Cheers, Pradyun -- > Distutils-SIG mailing list -- distutils-sig(a)python.org > To unsubscribe send an email to distutils-sig-leave(a)python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/distutils-sig@python.org/message/KKNR… >

1 0

distutils.util.get_host_platform() and AIX
by Michael 18 Aug '19

18 Aug '19

Recently I have become more interested in looking at how to simplify packaging for AIX. Currently I more or less ignore anything pip and/or distutils could mean for a python "user" (e.g., AIX sys admin and/or Python developer working on AIX) as I just run "pip install XYZ" locally, find what constitutes the equivalent of a bdist, and repackage those as 'installp' packages. For someone with root access this is ok - as installp requires root (or RBAC equivalent) to install. As I would like to make installing modules more Python like - I asked around and was pointed at piwheels (which looks very useful for my aspiration) but also at PEP425 and the platform-tag. There are several things I need to learn about dist-utils, and how packaging tagging needs to be considered when, e.g., Python is built on AIX 7.1 and the module is built on AIX 7.2 - there may be AIX ABI differences that would prevent the module built on AIX 7.2 from working properly on AIX 7.1 while the same module built on AIX 7.1 will run with no issue on AIX 7.2. (A simplified explanation is that libc.a on AIX 7.2 knows how to work with applications linked against libc.a from earlier versions of AIX, but earlier versions of AIX do not know how to deal with libc.a from AIX 7.2.) Taken to an extreme: Python(2|3) and modules built on AIX 5.3 TL7 run, unaltered, on all levels of AIX including the latest AIX 7.2 - while my expectation is that executable and modules built on AIX 7.2 TL5 (latest level) might not run on AIX 7.2 TL4. In short, "version", "release" are in themselves not enough - the TL (technology level) should also be considered. Additionally, what I also see "missing" is the platform.architecture()[0] value. By default, this is still 32bit on AIX - but it is important - especially for pre-built eggs and/or wheels. In the AIX world - the OS-level name scheme is usually: VR00-TL-SP (Version, Release, TechnologyLevel, ServicePack). There is also a value compareable to a builddate, but for distutils purposes has no value (I can think of). So, to my question: currently, for AIX get_host_platform returns something such as: aix-6.1. Considering above: what would you - as more expericenced with multi-oslevel packaging and low levels are accepted by high-levels, but not v.v. "What should the AIX get_host_platform() string contain?" At a minimum I forsee: return "%s-%s.%s-%s" % (osname, version, release, platform.architecture()[0]) But this does not address potential issues where the TL level within a version.release has changed. (X.Y.TL5 built packages MIGHT work on X.YTL4, but there is no reason to expect them to. So, I would look to something that remains recognizable, but uses different 'punctuation' e.g., oslevel -s returns a string such as: 6100-09-10-1731 Then using the equivalent of: version, release, service, builddata = '6100-09-10-1731'.split('-') return "%s-%s.%s.%s-%s" % (osname, version, release, service, platform.architecture()[0]) Note: no special authority is needed to run "oslevel -s", but it does take time. So having a way, in the library to only have to actually call for the value once - is a great improvement. I can imagine a way to do it (store a static value somewhere, and when it is NULL aka not initialized call the program, otherwise return the value) - but "WHERE" - in distutils, or (ideally?) elsewhere in a more "central" library. Sincerely, Michael

4 11