[Distutils] PEP 503 - Simple Repository API

[Donald Stufft]

On September 7, 2015 at 5:05:24 PM, M.-A. Lemburg (mal at egenix.com) wrote:
>
> In such a setup, the package manager would see lots and lots of
> links which are not relevant for the task. With the rel attributes,
> the package manager can focus on those links which are relevant.
> That's also the main reason for having those rel links in setuptools.

Well I mean, setuptools is still going to inspect each link with or without a
rel="download|homepage" to see if it "looks installable", which is the same
thing that pip does. I should probably call out explicitly that you cannot
assume that every single URL is a file though, that some may be unrelated.

If we were designing this from scratch I might agree with you, though I think
one of the biggest benefits of the "simple" repository is that it's extremely
simple for someone to get started with it, a file system and basically any 
web server is enough. So I'm not sure it's worth it to add the explicit rel
even if we were going from scratch if we lose that property.

>  
> Hmm, that's convention based and does not allow detecting
> the presence of such signatures without actually trying a download.
>  
> I think it would be better to make the availability explicit
> by adding an attribute to the link element (just like for other
> such features).

I'm OK with adding the attribute to links, though we should still mandate the
location. Neither pip nor setuptools will do anything with the PGP signatures
but some other tooling might. The legacy behavior of "just try the link" will
still work then, and if someone wants to do it more efficiently the attribute
is there. I'm not sure it's going to be generally useful since the signing on
PyPI doesn't really have a coherent threat model so it doesn't really protect
against much.

>  
> A dynamic API can be added as addition, but is hardly ever required
> for installation.

I think I've misled you!

I have no plans to make the installer API dynamic. I want as many requests as
possible to be served out of the Fastly cache instead of hitting the backends
as possible which means whatever we end up with will focus on static responses
and expecting the clients to do more requests and handle more things on the
client side rather than just querying the server for it.

However, I don't think that HTML is a very good data serialization format, it's
primary purpose is for document markup which isn't really the same thing as
data serialization. One such shortcoming is the total lack of any real data
type except for strings, looking at just the "has gpg signature" thing from
above, it'd be nicer if that could be modeled as:

    [
        {"url": "/path/to/filename.tar.gz", "has_sig": True}
    ]

Instead of relying on the presence or absence of a value (which doens't work
if you need to detect the difference between False/None) or using special
values that can be converted to some other data type if you know ahead of time
you need to do that for that particular value.

I don't know for a fact it'll be JSON, but I think it's an obvious choice given
that it's in the standard library, it's a reasonable data serialization format,
and it's human readable. When it comes to that time I think it'd be reasonable
to explore other formats to see if they make sense too, but whatever it will
be it's my intent it'll be static.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA