[Distutils] PyPI and GPG Signatures

[Donald Stufft]

> On May 12, 2016, at 8:56 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 12 May 2016 at 21:41, Donald Stufft <donald at stufft.io> wrote:
>> Thus, I would like to remove this feature from PyPI (but not from PEP 503, if
>> other repositories want to continue to support it they are free to). Doing this
>> would allow simplifying code we have in Warehouse anyplace we touch uploaded
>> files (since we almost always end up needing to branch into special behavior
>> for files ending with .asc). It will allow us to reduce the number of concepts
>> in the UI (what is a pgp signature? What do I do with it? etc) without simply
>> hiding a feature (which is likely to cause confusion, why do you support it if
>> you won't show it etc). I think it will also make releasing slightly easier for
>> developers, since I personally know a number of authors on PyPI who don't
>> really believe there is any value in signing their packages on PyPI, but they
>> do it anyways because of a vague notion that they should do it.
> 
> I'm a fan of most ideas that lower barriers to migration from the
> legacy PyPI codebase to Warehouse :)
> 
> However, I think one of the core points here is that even if GPG keys
> are removed from the *public* API, we still don't want the migration
> to break things for folks with automated upload processes that supply
> signature information - there'll be enough teething problems for the
> migration without a potential coincident breakage for a couple of
> thousands projects. That means the trade-off to consider is whether or
> not dropping this feature will get to the point of being able to
> deploy Warehouse sooner or not.


Both PyPI and Warehouse silently ignore additional fields so we get this for
free unless we go out of our way to add a check for the existence of it in the
POST and manually raise an error.

> 
> - if it's already implemented & tested in Warehouse, then leave it
> alone - you can take it out later, some time *after* the migration
> - if it's implemented, but not well tested, decide what to do based on
> whether testing or the below idea seems like less work
> - if it's not implemented in Warehouse yet, that's when the key
> requirement becomes "don't break automated signed uploads”

It is implemented for uploads (and like all code in Warehouse it has 100%
coverage) and I've had people successfully upload signatures to Warehouse with
it... but then I've also had "weird" errors where people attempt to upload a
signature to Warehouse and the `request.POST["gpg_signature"]` is a completely
different type of object than it normally is.

I have no idea *why* it was suddenly a different object for those people and
that comes from ``cgi.py`` in the standard library which is... well not super
easy to follow what's going on. So I'm moderately concerned about what is going
to happen if more people start trying to use that, particularly since I can't
really decipher what what is triggering this behavior to know how to guard
against it (other than just blindly writing the code to accept the additional
type I've observed with no way to reproduce the error and just hope that it's
enough to fix it). To make it worse, the problem was persistent until the
person in questioned reinstalled twine and it's dependencies (but I can't repro
with any version of twine).

It's not implemented the UI nor is the ability to associate a GPG key with a
particular user implemented nor is the ability to see what GPG key a user has
associated with their account implemented, and I sort of don't want to
implement those for all the reasons I listed in my original post. To do that
would require Nicole's time to properly design it, and her time is more limited
than mine is, so if I can reduce the things she needs to do by putting in some
effort on my part (like removing the code that's there, or implementing your
mail solution) that seems like a good trade off to me. Some folks who uploaded
GPG keys to Warehouse noticed they didn't show up in the UI and asked about it
(since they had uploaded them) and they felt that either Warehouse should fully
implement the GPG feature or it should get rid of it, but a half implementation
isn't great for anyone.

> 
> If we drop the feature, then my suggestion for a future upload UX is
> to let uploads with signatures succeed, but implicitly discard the
> signatures, and *automatically email the maintainer explaining why
> their uploaded signatures aren't appearing*.

That seems reasonable.

> 
> Similarly, projects that *had* signatures uploaded should gain a
> database flag indicating previously uploaded keys have been discarded,
> such that:
> 
> 1. The admin page for the project explains why the signatures are gone
> 2. The affected package maintainers are emailed with a list of
> projects they maintain from which signatures have been removed
> 
> The aim here would be to ensure that package maintainers tempted to
> ask "Where did my package signatures go?" are able to have that
> question answered *within the context of the service itself*, even if
> they've never heard of distutils-sig.

That seems reasonable to, and I don’t even really need to add a new
database column for it (though it might make sense for performance
sake) but this should be trivially query able using:

    SELECT EXISTS(
        SELECT 1
        FROM release_files
        WHERE name = %s AND has_signature = 't'
    );

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160512/e33e4320/attachment-0001.sig>