Distutils-SIG December 2018

distutils-sig@python.org

17 participants
11 discussions

by Chris Barker

Hi all, PyPi does not allow duplicate file names -- this makes lots of sense, because you really don't want people to go to PyPi one day and grab a file, and then go there another day, grab a file with exactly the same name, and have it be a different file. However.... We are all too human, and make mistakes when doing a release. All to often someone pushed a broken file up to PyPi, often realizes it pretty quickly -- before anyone has a chance to even download it (or only the dev team as, for testing...). In fact, I was in a sprint last summer, and we decided to push our package up to PyPi -- granted, we were all careless amateurish noobs, but we ended up making I think 4! minor version bumps because we had done something stupid in the sdist. Also -- the latest numpy release got caught in this, too: """ * We ran into a problem with pipy not allowing reuse of filenames and a resulting proliferation of *.*.*.postN releases. Not only were the names getting out of hand, some packages were unable to work with the postN suffix. """ So -- I propose that PyPi allow projects to replace existing files if they REALLY REALLY want to. You should have to jump through all sorts of hoops, and make it really clear that it is a BAD IDEA in the general case, but it'd be good to have it be possible. After all -- PyPi does not take on responsibility for anything else about what's in those packages, and Python itself is all about "we're all consenting adults here" I suppose we could even put in some heuristics about how long the file as been there, how many times it's been downloaded, etc. Just a thought.....I really hate systems that don't let me roll back mistakes, even when I discover them almost immediately... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 month

ImportError running 'test' on namespace package - other package in namespace not found

by Jason R. Coombs

I've got two projects: mynamespace.myprojectA and mynamespace.myprojectB myprojectB depends on myprojectA. I'm using setuptools 0.6c8 to manage both projects. Both projects are registered using 'setup develop'. Both projects are accessible from an interactive interpreter: PS C:\Users\me\projects> python Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import mynamespace.myprojectA >>> import mynamespace.myprojectB >>> from mynamespace.myprojectA import mymoduleZ However, when I run 'setup test' in myprojectB, the tests fail with File ".mymoduleZ.py", line NNN, in [some context] from mynamespace.myprojectA.mymoduleZ import MyClassC ImportError: No module named myprojectA.mymoduleZ In setup.py, the test_suite is nose.collector. I searched and couldn't find anyone else with this problem. Is this a supported configuration? Is there something I can do to make tests work with interdependent projects with the same root namespace? If there's not something obvious I should be doing differently, I'm happy to put together a minimal test case that reproduces the problem. Any suggestions are appreciated. Sincerely, Jason R. Coombs

1 month, 1 week

Surviving a Compromise of PyPI - PEP 458 and 480

by Vladimir Diaz

Hello everyone, I am a research programmer at the NYU School of Engineering. My colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting community feedback on our proposal, "Surviving a Compromise of PyPI." The two-stage proposal can be reviewed online at: PEP 458 http://legacy.python.org/dev/peps/pep-0458/ PEP 480 http://legacy.python.org/dev/peps/pep-0480/ Summary of the Proposal: "Surviving a Compromise of PyPI" proposes how the Python Package Index (PyPI) can be amended to better protect end users from altered or malicious packages, and to minimize the extent of PyPI compromises against affected users. The proposed integration allows package managers such as pip to be more secure against various types of security attacks on PyPI and defend end users from attackers responding to package requests. Specifically, these PEPs describe how PyPI processes should be adapted to generate and incorporate repository metadata, which are signed text files that describe the packages and metadata available on PyPI. Package managers request (along with the packages) the metadata on PyPI to verify the authenticity of packages before they are installed. The changes to PyPI and tools will be minimal by leveraging a library, The Update Framework <https://github.com/theupdateframework/tuf>, that generates and transparently validates the relevant metadata. The first stage of the proposal (PEP 458 <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model that supports verification of PyPI packages signed with cryptographic keys stored on PyPI, requires no action from developers and end users, and protects against malicious CDNs and public mirrors. To support continuous delivery of uploaded packages, PyPI administrators sign for uploaded packages with an online key stored on PyPI infrastructure. This level of security prevents packages from being accidentally or deliberately tampered with by a mirror or a CDN because the mirror or CDN will not have any of the keys required to sign for projects. The second stage of the proposal (PEP 480 <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic security model (discussed in PEP 458) that supports end-to-end verification of signed packages. End-to-end signing allows both PyPI and developers to sign for the packages that are downloaded by end users. If the PyPI infrastructure were to be compromised, attackers would be unable to serve malicious versions of these packages without access to the project's developer key. As in PEP 458, no additional action is required by end users. However, PyPI administrators will need to periodically (perhaps every few months) sign metadata with an offline key. PEP 480 also proposes an easy-to-use key management solution for developers, how to interface with a potential build farm on PyPI infrastructure, and discusses the security benefits of end-to-end signing. The second stage of the proposal simultaneously supports real-time project registration and developer signatures, and when configured to maximize security on PyPI, less than 1% of end users will be at risk even if an attacker controls PyPI and goes undetected for a month. We thank Nick Coghlan and Donald Stufft for their valuable contributions, and Giovanni Bajo and Anatoly Techtonik for their feedback. Thanks, PEP 458 & 480 authors.

3 months, 1 week

proposing Python package index upload API spec (potential PEP)

by Sumana Harihareswara

As a new Twine maintainer I've been running into questions like: * Now that Warehouse doesn't use "register" anymore, can we deprecate it from distutils, setuptools, and twine? Are any other package indexes or upload tools using it? https://github.com/pypa/twine/issues/311 * It would be nice if Twine could depend on a package index providing an HTTP 201 response in response to a successful upload, and fail on 200 (a response some non-package-index servers will give to an arbitrary POST request). I do not see specifications to guide me here, e.g., in the official guidance on hosting one's own package index https://packaging.python.org/guides/hosting-your-own-index/ . PEP 301 was long enough ago that it's due an update, and PEP 503 only concerns browsing and download, not upload. I suggest that I write a PEP specifying an API for uploading to a Python package index. This PEP would partially supersede PEP 301 and would document the Warehouse reference implementation. I would write it in collaboration with the Warehouse maintainers who will develop the reference implementation per pypa/warehouse/issues/284 and maybe add a header referring to compliance with this new standard. And I would consult with the maintainers of packaging and distribution tools such as zest.releaser, flit, poetry, devpi, pypiserver, etc. Per Nick Coghlan's formulation, my specific goal here would be close to: > Documenting what the current upload API between twine & warehouse actually is, similar to the way PEP 503 focused on describing the status quo, without making any changes to it. That way, other servers (like devpi) and other upload clients have the info they need to help ensure interoperability. Since Warehouse is trying to redo its various APIs in the next several months, I think it might be more useful to document and work with the new upload API, but I'm open to feedback on this. After a little conversation here on distutils-sig, I believe my steps would be: 1. start a very early PEP draft with lots of To Be Determined blanks, submit as a PR to the python/peps repo, and share it with distutils-sig 2. ping maintainers of related tools 3. discuss with others at the packaging sprints https://wiki.python.org/psf/PackagingSprints next week 4. revise and get consensus, preferably mostly on this list 5. finalize PEP and get PEP accepted by BDFL-Delegate 6. coordinate with PyPA, maintainers of `distutils`, maintainers of packaging and distribution tools, and documentation maintainers to implement PEP compliance Thoughts are welcome. I originally posted this at https://github.com/pypa/packaging-problems/issues/128 . -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

6 months, 1 week

PyPI will no longer accept compromised passwords!

by Donald Stufft

I just wanted to give everyone a heads up that PyPI will no longer accept passwords that have been published in data breaches. For background you can take a look at https://github.com/pypa/warehouse/pull/4541 <https://github.com/pypa/warehouse/pull/4541>. For high level overview see https://pypi.org/help/#compromised-password <https://pypi.org/help/#compromised-password>. Finally if you have any trouble, please file an issue at https://github.com/pypa/warehouse/issues/new/choose <https://github.com/pypa/warehouse/issues/new/choose>.

11 months, 1 week

Idea: perennial manylinux tag

by Nathaniel Smith

Hi all, The manylinux1 -> manylinux2010 transition has turned out to be very difficult. Timeline so far: March 2017: CentOS 5 went EOL April 2018: PEP 517 accepted May 2018: support for manylinux2010 lands in warehouse November 2018: support lands in auditwheel, and pip master December 2018: 21 months after CentOS 5 EOL, wwee still don't have an official build environment, or support in a pip release We'll get through this, but it's been super painful and maybe we can change things somehow so it will suck less next time. We don't have anything like this pain on Windows or macOS. We never have to update pip, warehouse, etc., after those OSes hit EOLs. Why not? On Windows, we have just two tags: "win32" and "win_amd64". These are defined to mean something like "this wheel will run on any recent-ish Windows system". So the meaning of the tag actually changes over time: it used to be that if a wheel said it ran on win32, then that meant it would work on winxp, but since winxp hit EOL people started uploading "win32" wheels that don't work on winxp, and that's worked fine. On macOS, the tags look like "macosx_10_9_x86_64". So here we have the OS version embedded in the tag. This means that we do occasionally switch which tags we're using, kind of like how manylinux1 -> manylinux2010 is intended to work. But, unlike for the manylinux tags, defining a new macosx tag is totally trivial: every time a new OS version is released, the tag springs into existence without any human intervention. Warehouse already accepts uploads with this tag; pip already knows which systems can install wheels with this tag, etc. Can we take any inspiration from this for manylinux? We could do the Windows thing, and have a plain "manylinux" tag that means "any recent-ish glibc-based Linux". Today it would be defined to be "any distro newer than CentOS 6". When CentOS 6 goes out of service, we could tweak the definition to be "any distro newer than CentOS 7". Most parts of the toolchain wouldn't need to be updated, though, because the tag wouldn't change, and by assumption, enforcement wouldn't really be needed, because the only people who could break would be ones running on unsupported platforms. Just like happens on Windows. We could do the macOS thing, and have a "manylinux_${glibc version}" tag that means "this package works on any Linux using glibc newer than ${glibc version}". We're already using this as our heuristic to handle the current manylinux profiles, so e.g. manylinux1 is effectively equivalent to manylinux_2_5, and manylinux2010 will be equivalent to manylinux_2_12. That way we'd define the manylinux tags once, get support into pip and warehouse and auditwheel once, and then in the future the only thing that would have to change to support new distro releases or new architectures would be to set up a proper build environment. What do y'all think? -n

1 year, 2 months

PEP 440

by Michael Goerz

PEP 440 contains a section that claims: > Summary of differences from pkg_resources.parse_version [https://www.python.org/dev/peps/pep-0440/#id63] > > * Local versions sort differently, this PEP requires that they sort as greater than the same version without a local version, whereas pkg_resources.parse_version considers it a pre-release marker. I haven't been able to find any mention of this in the setuptools changelogs, but this no longer seems to be the case: >>> from pkg_resources import parse_version >>> parse_version('1.0.0+dev') > parse_version('1.0.0') True Thus, it might be good to change the comment in the PEP accordingly (ideally, someone is able to figure out in which version of setuptools parse_version started to be compatible with PEP440). Best, Michael Goerz

1 year, 4 months

PEP 541

by Elliot Paton-Simpson

Hello, I am currently developing an application called "Outbreak" and would like to release a package to Pypi. However, there already exists a package with that name <https://pypi.org/project/outbreak/>. It has no description, has no functionality, and hasn't been updated in four months. According to PEP 541, such a project is considered name squatting and is thus to be considered invalid. May I please request that the project be removed from the Package Index? If you would like, I can contact the author myself. Thank you, Elliot Paton-Simpson

1 year, 4 months

Announcing dotlock - another package management tool

by Alex Becker

Dotlock <https://pypi.org/project/dotlock/> is a package management tool similar to pipenv, but with a different philosophy: instead of acting as a wrapper around pip, dotlock handles package resolution natively. This should allow for more flexibility, better performance, and a smaller surface area for bugs, but at the cost of the wide platform and package support the pip developers have put so much work into. Dotlock is still very much beta software, and currently the only users are myself and my company. It works well for our use case (deploying web applications to linux servers), but has some gaps in support I know about (including packages that use setup_requires or pyproject.toml), and probably many I do not. Thanks to everyone on this list who I discussed dotlock and general package issues with at PyCon, and to anyone who tries it out! Sincerely, Alex Becker

1 year, 4 months

upcoming work to facilitate PyPA communications/roadmaps

by Sumana Harihareswara

It's been eight months since the release of Warehouse[0] and the sunsetting of legacy PyPI[1]. Following up from our meeting at PyCon in May[2], Changeset Consulting is back on board for another round of project management to facilitate next steps! For the next 3-6 months this work will be spearheaded by myself (Sumana) assisted by Jenny Ryan (https://jennyryan.net ). The goal over these upcoming months is to create, steward and facilitate internal and public-facing communications to aid the folks within PyPA. What this means is that we'll be focused on the following: * Facilitating regular meetings of and for maintainers and contributors; * Stewarding communications with various PyPA stakeholders, including funders and users; * Organizing, labelling, prioritizing, and responding to GitHub issues; * Coordinating public communications, such as announcements, sprints, and calls for participation; * Maintaining and improving documentation, meeting notes and development roadmaps for PyPA projects. Feedback from and participation by the Python packaging developer community is obviously part and parcel of this project, so you may see some new "here's what I think is up with this issue, is that right?" questions on old unresolved discussions. And we'll be asking questions on this & other lists and on GitHub and in IRC to collect ideas, concerns, and other productive input regarding the tools roadmaps. You'll be seeing more details in mid-January to properly kick off this next chapter of levelling up PyPI and the PyPA -- just wanted to give y'all a heads-up. But of course, if you were already planning on using the next few weeks to do issue triage and roadmap-writing and PyCon planning, please don't wait for us -- that'll make this work all the easier. Thanks, Sumana Harihareswara [0] https://blog.python.org/2018/04/new-pypi-launched-legacy-pypi-shutting.html [1] https://mail.python.org/archives/list/distutils-sig@python.org/thread/YREMU… [2] https://mail.python.org/archives/list/distutils-sig@python.org/thread/CCOV6… -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 year, 5 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

Distutils-SIG December 2018