Distutils-SIG August 2019

distutils-sig@python.org

20 participants
16 discussions

by Chris Barker

Hi all, PyPi does not allow duplicate file names -- this makes lots of sense, because you really don't want people to go to PyPi one day and grab a file, and then go there another day, grab a file with exactly the same name, and have it be a different file. However.... We are all too human, and make mistakes when doing a release. All to often someone pushed a broken file up to PyPi, often realizes it pretty quickly -- before anyone has a chance to even download it (or only the dev team as, for testing...). In fact, I was in a sprint last summer, and we decided to push our package up to PyPi -- granted, we were all careless amateurish noobs, but we ended up making I think 4! minor version bumps because we had done something stupid in the sdist. Also -- the latest numpy release got caught in this, too: """ * We ran into a problem with pipy not allowing reuse of filenames and a resulting proliferation of *.*.*.postN releases. Not only were the names getting out of hand, some packages were unable to work with the postN suffix. """ So -- I propose that PyPi allow projects to replace existing files if they REALLY REALLY want to. You should have to jump through all sorts of hoops, and make it really clear that it is a BAD IDEA in the general case, but it'd be good to have it be possible. After all -- PyPi does not take on responsibility for anything else about what's in those packages, and Python itself is all about "we're all consenting adults here" I suppose we could even put in some heuristics about how long the file as been there, how many times it's been downloaded, etc. Just a thought.....I really hate systems that don't let me roll back mistakes, even when I discover them almost immediately... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

2 months, 3 weeks

ImportError running 'test' on namespace package - other package in namespace not found

by Jason R. Coombs

I've got two projects: mynamespace.myprojectA and mynamespace.myprojectB myprojectB depends on myprojectA. I'm using setuptools 0.6c8 to manage both projects. Both projects are registered using 'setup develop'. Both projects are accessible from an interactive interpreter: PS C:\Users\me\projects> python Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import mynamespace.myprojectA >>> import mynamespace.myprojectB >>> from mynamespace.myprojectA import mymoduleZ However, when I run 'setup test' in myprojectB, the tests fail with File ".mymoduleZ.py", line NNN, in [some context] from mynamespace.myprojectA.mymoduleZ import MyClassC ImportError: No module named myprojectA.mymoduleZ In setup.py, the test_suite is nose.collector. I searched and couldn't find anyone else with this problem. Is this a supported configuration? Is there something I can do to make tests work with interdependent projects with the same root namespace? If there's not something obvious I should be doing differently, I'm happy to put together a minimal test case that reproduces the problem. Any suggestions are appreciated. Sincerely, Jason R. Coombs

2 months, 3 weeks

Surviving a Compromise of PyPI - PEP 458 and 480

by Vladimir Diaz

Hello everyone, I am a research programmer at the NYU School of Engineering. My colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting community feedback on our proposal, "Surviving a Compromise of PyPI." The two-stage proposal can be reviewed online at: PEP 458 http://legacy.python.org/dev/peps/pep-0458/ PEP 480 http://legacy.python.org/dev/peps/pep-0480/ Summary of the Proposal: "Surviving a Compromise of PyPI" proposes how the Python Package Index (PyPI) can be amended to better protect end users from altered or malicious packages, and to minimize the extent of PyPI compromises against affected users. The proposed integration allows package managers such as pip to be more secure against various types of security attacks on PyPI and defend end users from attackers responding to package requests. Specifically, these PEPs describe how PyPI processes should be adapted to generate and incorporate repository metadata, which are signed text files that describe the packages and metadata available on PyPI. Package managers request (along with the packages) the metadata on PyPI to verify the authenticity of packages before they are installed. The changes to PyPI and tools will be minimal by leveraging a library, The Update Framework <https://github.com/theupdateframework/tuf>, that generates and transparently validates the relevant metadata. The first stage of the proposal (PEP 458 <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model that supports verification of PyPI packages signed with cryptographic keys stored on PyPI, requires no action from developers and end users, and protects against malicious CDNs and public mirrors. To support continuous delivery of uploaded packages, PyPI administrators sign for uploaded packages with an online key stored on PyPI infrastructure. This level of security prevents packages from being accidentally or deliberately tampered with by a mirror or a CDN because the mirror or CDN will not have any of the keys required to sign for projects. The second stage of the proposal (PEP 480 <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic security model (discussed in PEP 458) that supports end-to-end verification of signed packages. End-to-end signing allows both PyPI and developers to sign for the packages that are downloaded by end users. If the PyPI infrastructure were to be compromised, attackers would be unable to serve malicious versions of these packages without access to the project's developer key. As in PEP 458, no additional action is required by end users. However, PyPI administrators will need to periodically (perhaps every few months) sign metadata with an offline key. PEP 480 also proposes an easy-to-use key management solution for developers, how to interface with a potential build farm on PyPI infrastructure, and discusses the security benefits of end-to-end signing. The second stage of the proposal simultaneously supports real-time project registration and developer signatures, and when configured to maximize security on PyPI, less than 1% of end users will be at risk even if an attacker controls PyPI and goes undetected for a month. We thank Nick Coghlan and Donald Stufft for their valuable contributions, and Giovanni Bajo and Anatoly Techtonik for their feedback. Thanks, PEP 458 & 480 authors.

4 months, 4 weeks

proposing Python package index upload API spec (potential PEP)

by Sumana Harihareswara

As a new Twine maintainer I've been running into questions like: * Now that Warehouse doesn't use "register" anymore, can we deprecate it from distutils, setuptools, and twine? Are any other package indexes or upload tools using it? https://github.com/pypa/twine/issues/311 * It would be nice if Twine could depend on a package index providing an HTTP 201 response in response to a successful upload, and fail on 200 (a response some non-package-index servers will give to an arbitrary POST request). I do not see specifications to guide me here, e.g., in the official guidance on hosting one's own package index https://packaging.python.org/guides/hosting-your-own-index/ . PEP 301 was long enough ago that it's due an update, and PEP 503 only concerns browsing and download, not upload. I suggest that I write a PEP specifying an API for uploading to a Python package index. This PEP would partially supersede PEP 301 and would document the Warehouse reference implementation. I would write it in collaboration with the Warehouse maintainers who will develop the reference implementation per pypa/warehouse/issues/284 and maybe add a header referring to compliance with this new standard. And I would consult with the maintainers of packaging and distribution tools such as zest.releaser, flit, poetry, devpi, pypiserver, etc. Per Nick Coghlan's formulation, my specific goal here would be close to: > Documenting what the current upload API between twine & warehouse actually is, similar to the way PEP 503 focused on describing the status quo, without making any changes to it. That way, other servers (like devpi) and other upload clients have the info they need to help ensure interoperability. Since Warehouse is trying to redo its various APIs in the next several months, I think it might be more useful to document and work with the new upload API, but I'm open to feedback on this. After a little conversation here on distutils-sig, I believe my steps would be: 1. start a very early PEP draft with lots of To Be Determined blanks, submit as a PR to the python/peps repo, and share it with distutils-sig 2. ping maintainers of related tools 3. discuss with others at the packaging sprints https://wiki.python.org/psf/PackagingSprints next week 4. revise and get consensus, preferably mostly on this list 5. finalize PEP and get PEP accepted by BDFL-Delegate 6. coordinate with PyPA, maintainers of `distutils`, maintainers of packaging and distribution tools, and documentation maintainers to implement PEP compliance Thoughts are welcome. I originally posted this at https://github.com/pypa/packaging-problems/issues/128 . -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

7 months, 3 weeks

Linux binary wheels?

by Dan Stromberg

Hi folks. I have a pair of ideas about Linux binary wheels, which are currently (I heard) unsupported. It seems like it should be possible to support Linux binary wheels using one or both of these technologies: * https://build.opensuse.org/ is a service that builds packages for a variety of Linuxes * Docker could be used to automate the building of wheels for a handful of Linuxes with minimal dependencies. It seems like if you get Debian/Ubuntu/Mint, Fedora/CentOS, openSUSE and perhaps one or two others, that would cover almost all Linuxes and Linux users. I'm up to my hears in commitments already, but I sincerely someone will grab onto one or both of these possibilities and run with them. Thanks for reading.

10 months, 2 weeks

timeline for new pip resolver rollout - feedback?

by Sumana Harihareswara

What timeline are we thinking is realistic for rolling out the new pip resolver? (latest update on resolver work: https://pradyunsg.me/blog/2019/08/06/pip-update-2/ ) I'm re-upping this question which I originally asked on a GitHub issue about the rollout: https://github.com/pypa/pip/issues/6536#issuecomment-521696430 and would prefer to corral answers there. This depends a lot on Pradyun's health and free time, and code review availability from other pip maintainers, and whether we get some grants we're applying for, but I think the sequence is something like: 1) build logic refactor: in progress, done sometime December-February 2) UX research and design, test infrastructure building, talking to downstreams and users about config flags and transition schedules: we need funding for this; earliest start is probably December, will take 2-3 months 3) introduce the abstractions defined in resolvelib/zazo while doing alpha testing: will take a few months, so, conservatively estimating, May 2020? 4) adopting better dependency resolution and do beta testing: ? Is this right? What am I missing? I ask because some of the info-gathering work is stuff a project manager and/or UX researcher should do, in my opinion, and because some progress on the increase in metadata strictness https://github.com/pypa/packaging-problems/issues/264 and other issues might help with concerns people have brought up here. -- Sumana Harihareswara PyPI project manager, PyPA member & coordinator, and person who seems to write a lot of grant applications Changeset Consulting https://changeset.nyc

10 months, 2 weeks

pip and PEP 517 Backends

by Phil Thompson

I have a PEP 517 compatible backend which works with pip to install from an sdist (via an internal wheel). However there are a couple of issues... pip swallows all output from the backend. Is there anyway for the user to see the output (builds can take several minutes)? I would like to pass options from the pip command line to the backend but neither --global-option or --install-option have any effect (the config_settings argument is always None). How do I achieve this? Thanks, Phil

10 months, 2 weeks

Adding support for adequately tagging AIX (pep425) to support distributed wheels

by Michael

Goal - using wheels rather than RPM and/or installp formats for distributing binary modules Why bother? One reason (there are likely more) - using wheels means packages/modules can be loaded in a virtualenv rather than require they are first loaded in the system environment using installp/rpm/yum (with root authority). This alone has been reason enough for me to do the research. Introduction The use of pip and wheels is commonplace in the worlds of Linux, macOS and Windows. Not so for AIX. Not because it couldn't be commonplace. The current situation for AIX is comparable to the initial issue Linux was facing when PEP513 was written: "Currently, distribution of binary Python extensions for Windows and OS X is straightforward. ... For Linux, the situation is much more delicate. ... Build tools using PEP 425 platform tags [3] do not track information about the particular Linux distribution or installed system libraries, and instead assign all wheels the too-vague linux_i686 or linux_x86_64 tags. Because of this ambiguity, ..." The root cause for the *ambiguity* that Linux systems had is not the ambiguity that AIX faces. AIX has provided a consistent way to "tag" it's runtime environment since at least 2007 (AIX 5.3 TL7). Since that time IBM AIX has also *guaranteed* binary compatibility for migration of applications from old to new OS levels. I would like to see these tags added - at a minimum that they can be retrieved by something such as sysconfig.get_var('AIC_BLD_TAG'). It would be "nice" to see sysconfig.get_platform() updated to include these values from the running system. Further, while pip related tools can add the "bitness" to the platform tags I would like to see something added to the AIX get_platform() tag (b32, ppc, aix32), (b64, ppc64, aix64) for 32 and 64 bit operations, respectively - as that is a "running" environment attribute. Open to other ideas on what the bitness tag should be. IMHO - anything is better than nothing. Maybe this could be considered a bug rather than as a new feature. Thank you for your feedback. Michael

10 months, 3 weeks

Re: distutils.util.get_host_platform() and AIX

by Pradyun Gedam

Resending since I messed up the reply. On Sun, 18 Aug 2019 at 8:29 PM, Michael <aixtools(a)felt.demon.nl> wrote: > Would be 'nice' if old meant 3.6 and earlier as I would like to target 3.7 > and later. > I understand we're talking about generating the PEP 425 tags here. If not, please ignore this next paragraph. The "algorithm" essentially needs to be a check for "is this an AIX python" and also generate/provide a signal of what the system/binary compatibility looks like. This algorithm would executed on a target Python interpreter (which can be anything from Python 2.7, 3.3, 3.7 or a hypothetical 3.10), to check if that interpreter compatible with whatever tag you plan to generate. Reiterating - continue only here, or expand and include python-dev? > Continue here. python-dev has, practically, delegated Packaging discussions to this list. (opinion) If you're okay with changing communication channel, I have a strong preference for using discuss.python.org/c/packaging instead. Anyway, most of the packaging-related discussions have been happening there lately. Cheers, Pradyun -- > Distutils-SIG mailing list -- distutils-sig(a)python.org > To unsubscribe send an email to distutils-sig-leave(a)python.org > https://mail.python.org/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/archives/list/distutils-sig@python.org/message/KKNR… >

10 months, 3 weeks

distutils.util.get_host_platform() and AIX

by Michael

Recently I have become more interested in looking at how to simplify packaging for AIX. Currently I more or less ignore anything pip and/or distutils could mean for a python "user" (e.g., AIX sys admin and/or Python developer working on AIX) as I just run "pip install XYZ" locally, find what constitutes the equivalent of a bdist, and repackage those as 'installp' packages. For someone with root access this is ok - as installp requires root (or RBAC equivalent) to install. As I would like to make installing modules more Python like - I asked around and was pointed at piwheels (which looks very useful for my aspiration) but also at PEP425 and the platform-tag. There are several things I need to learn about dist-utils, and how packaging tagging needs to be considered when, e.g., Python is built on AIX 7.1 and the module is built on AIX 7.2 - there may be AIX ABI differences that would prevent the module built on AIX 7.2 from working properly on AIX 7.1 while the same module built on AIX 7.1 will run with no issue on AIX 7.2. (A simplified explanation is that libc.a on AIX 7.2 knows how to work with applications linked against libc.a from earlier versions of AIX, but earlier versions of AIX do not know how to deal with libc.a from AIX 7.2.) Taken to an extreme: Python(2|3) and modules built on AIX 5.3 TL7 run, unaltered, on all levels of AIX including the latest AIX 7.2 - while my expectation is that executable and modules built on AIX 7.2 TL5 (latest level) might not run on AIX 7.2 TL4. In short, "version", "release" are in themselves not enough - the TL (technology level) should also be considered. Additionally, what I also see "missing" is the platform.architecture()[0] value. By default, this is still 32bit on AIX - but it is important - especially for pre-built eggs and/or wheels. In the AIX world - the OS-level name scheme is usually: VR00-TL-SP (Version, Release, TechnologyLevel, ServicePack). There is also a value compareable to a builddate, but for distutils purposes has no value (I can think of). So, to my question: currently, for AIX get_host_platform returns something such as: aix-6.1. Considering above: what would you - as more expericenced with multi-oslevel packaging and low levels are accepted by high-levels, but not v.v. "What should the AIX get_host_platform() string contain?" At a minimum I forsee: return "%s-%s.%s-%s" % (osname, version, release, platform.architecture()[0]) But this does not address potential issues where the TL level within a version.release has changed. (X.Y.TL5 built packages MIGHT work on X.YTL4, but there is no reason to expect them to. So, I would look to something that remains recognizable, but uses different 'punctuation' e.g., oslevel -s returns a string such as: 6100-09-10-1731 Then using the equivalent of: version, release, service, builddata = '6100-09-10-1731'.split('-') return "%s-%s.%s.%s-%s" % (osname, version, release, service, platform.architecture()[0]) Note: no special authority is needed to run "oslevel -s", but it does take time. So having a way, in the library to only have to actually call for the value once - is a great improvement. I can imagine a way to do it (store a static value somewhere, and when it is NULL aka not initialized call the program, otherwise return the value) - but "WHERE" - in distutils, or (ideally?) elsewhere in a more "central" library. Sincerely, Michael

10 months, 4 weeks

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

Distutils-SIG August 2019