[Distutils] Name arbitration on PyPI (was: The mypy package)

[Donald Stufft]

> On Apr 18, 2016, at 5:16 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> 
> 1.  PyYAML is a package that would be de-registered in such a scheme.  It is a highly used, extremely popular, package that unserializes text into arbitrary python objects.  It is a trusted package... and one that hasn't been active in ages.
> 
> and you don't think ANYONE would be willing to take on the miniscule amount of work to maintain the name? Plus there would be any number of other schemes for determining whether a project name is abandoned.


This sort of gets to the core of one of the major questions that has to be answered before you talk about expiring names and such. Who “owns” a name on PyPI? Right now, as implemented, the people who “own” (in terms of ACL) the name on PyPI are the owners of said name. In this vein PyPI has traditionally been first come first serve on names and generally will not release a name to someone else without the current owner’s permission [1]. This leads to projects, even popular ones, sometimes falling into abandonment, sometimes even when you have a new maintainer willing to step up (as is the case with Ian and pyYAML) or in some cases, has already forked the project under another name (as is the case with PIL and Pillow). However, this also means that in general, if you decide you trust the current owner of something on PyPI, you know that it’s not suddenly going to be owned by someone else *unless* the person who you previously trusted decides to give it away to them (and thus, implicitly transferring the trust you granted them to another person).

So, before we try and think up some mechanism for expiring names, or forcibly taking a name from the current owner and giving it to another, we [2] would first need to decide that names on PyPI are not owned by the individuals who hold the registration, but are instead owned by the community and the current individuals are only borrowing them. This would mean that instead of having packages wither and get abandoned we could instead transfer them to new maintainers who can keep the project going. However, that of course raises new questions like:

* Should we only transfer projects to maintainers who want to keep the current project going? Such as with PIL or pyYAML?
  - This will be generally less surprising to end users, but it doesn’t solve the problem of rarely used packages “blocking” other’s use of the name.

* How do we determine who to give a name to? Surely we shouldn’t hand off a package that people are using to the first random passerby who happens to ask for it and  we can probably trust Guido not to be malicious (or else we have bigger problems ;) ), but how do we determine if someone is to be trusted when they inevitably fall somewhere in between?

* The current policy is pretty black and white with only a little bit of grey area (what is a “real” package that’s been uploaded to a project, what is a malicious package, etc), but moving over to community owned names would open things up to a far wider set of grey areas, and who is going to arbitrate when a project is abandoned or when it’s “popular” or not?
  - If folks are unaware, npmjs.org <http://npmjs.org/> just recently had a bit of a kerfuffle over a package on their repository going missing, “left-pad”, due to the grey-ness of their policy.
  - The current, fairly rigid policy came around because the PyPI admins decided to give what appeared to be an unmaintained library to someone new who had forked the library, after the original author hadn’t responded to an email. This ended up causing our own kerfuffle when it turned out this person *wasn’t* absent and *wasn’t* OK with this. [3].

* If we start giving projects out to other people, what tools should we give people to deal with that?
  - The latest version of pip has a mode that allows you to specify a number of hashes and it won’t install anything that doesn’t match a hash. That could protect end users from installing something inadvertently, but it’s very opt in and requires hard pins ``==`` so it’s not likely to get a large uptick.
  - If we mandated semver (or something like it) we could make it so that transferring a name *forced* a major version bump and the new author would be unable to release anything using a smaller major version number, people could then pin to <CURRENT_MAJOR+1 and be assured that somebody new won’t take over a project without their knowing about it. However we haven’t historically mandated this and there are a lot of projects using date based versions that would b very unhappy (in addition to the fact upper pins often are major contributors to being unable to resolve a version tree of dependencies).
  - If we implemented a signature scheme, we could possibly flag certain key changes as must-require-confirmation or something like it and if users have previously used the old key, they won’t install the new package without first confirming the key change.

Anyways, the very first thing we need to decide is if projects are community or individual owned, and if community owned, under what situations are we willing to transfer and what stipulations there should be. Once we have all that, figuring out a mechanism becomes a lot easier (even if that mechanism is manual).

[1] The main exception to this rule is when the name is obviously being used for nothing and the current owner is not reachable by one of the PyPI admins. In this case we’ve vacated a name and assigned it to someone else without the current owner’s OK. Other exceptions are things like malicious packages, spam, etc.
[2] Who is we? To be honest, I don’t really know. I suspect this is something that could be handled as a PEP perhaps, or maybe this is something that the PSF Board should decide.
[3] The whole thing was a bit more complicated than that, there were some process failures on our side as well.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160418/4879e66a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160418/4879e66a/attachment-0001.sig>