[Distutils] PEP 503 - Simple Repository API

[Donald Stufft]

On September 5, 2015 at 5:43:58 AM, M.-A. Lemburg (mal at egenix.com) wrote:
>  
> Hmm, if the installer will build the URL itself, why is there even
> a need for a top-level index page ?
>  
> I mean for the occasional human reading the page it will certainly
> make sense to have such a page, but for the API this doesn't
> appear to be essentially needed.
>  
> Or is the idea to have the package manager scan the index for package
> hosted on that index prior to asking for the package it would like
> to install ?

The latest versions of pip won't use it, setuptools and older versions of pip
will use it though. The versions of pip/setuptools that would use it, use it as
a fallback. They don't pre-normalize the name before requesting the URL so they
just used whatever the user typed. This comes from when a project like "Django"
was at /simple/Django/ on PyPI but if a user typed ``pip install django`` it
would first fetch /simple/django/ and if that 404'd it would fall back onto
/simple/ and look for these links. On PyPI this rarely happened because PyPI
redirects /simple/anything-that-normalizes-to-the-name/ to the correct URL but
it's useful for static repositories that don't have something to redirect it
in front.

I've tried to make it so that all of the SHOULD and MUST directives can be
implement by a standard Nginx/Apache/whatever web server with static files
while maintaining compatability with older installers.

>  
> Would it help the package manager to more easily detect the links
> that point to distribution files instead of e.g. documentation or
> other resources ?
>  
> setuptools uses rel="download" for this:
>  
> https://pythonhosted.org/setuptools/easy_install.html#package-index-api 

This is actually for the link spidering that PEP 470 removed, links marked with
either rel="download" or rel="homepage" would be fetched (unless they looked
installable) and searched for additional links before PEP 438/470 started to
deprecate/remove them. Both setuptools and pip only need a simple page that has
links that point to files on the, see for example the /simple/ page for
requests: https://pypi.python.org/simple/requests/

>  
> Could we perhaps also add optional features like:
>  
> * Distribution link elements MAY include a data-gpg-sig=""
> attribute to provide a GPG signature of the linked file
>  
> This could later be extended to more meta data, such as platform
> tags, distribution file types, license info, mirror locations,
> documentation, help strings, etc.

I actually forgot to mention the GPG signatures, currently the assumption is
that if a GPG signature exists it will live at the same location as the file
with a .asc on the end, so if the file is /packages/Django-1.0.tar.gz then the
GPG signature will be located at /packages/Django-1.0.tar.gz.asc. I'll add this
to the PEP.

I don't want to add more features to the API, particularly not in this PEP. My
longer term plan is to work on a a new API for installers to use which will be
easier to work with. The current API is great for it's simplicity and the fact
it can be implemented on the server side with nothing more than a directory
structure full of files and python -m http.server. The plan in my head is to
add a new repository API which can handle the more complex cases AND which will
most likely be JSON based to simplify parsing of it. The simple API would not
be deprecated, it would just be up to the repository which "version" of the API
they use. For people hosting their own repositories, if they have a simple case
they will be able to get away with the simple API, but the more complex API
would offer benefits like being able to access the metadata information without
downloading the file.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA