[Distutils] The Simple API - What URLs are "supported"
Donald Stufft
donald at stufft.io
Thu Sep 18 12:25:37 CEST 2014
> On Sep 18, 2014, at 3:27 AM, holger krekel <holger at merlinux.eu> wrote:
>
> On Wed, Sep 17, 2014 at 20:59 -0400, Donald Stufft wrote:
>> Right now pip (and originally setuptools, which does it as well) will do this
>> sort of dance when looking for things on the PyPI simple index. This isn't the
>> actual code though:
>>
>> thing_to_install = "foo==1.0"
>> page = None
>>
>> if thing_to_install.contains("=="): # First look at a versioned url if ==
>> page = request_url(
>> "https://pypi.python.org/simple/" + thing_to_install.name
>> + "/" + thing_to_install.version
>> )
>>
>> if not page: # If we don't have something, look for unversioned
>> page = request_url(
>> "https://pypi.python.org/simple/" + thing_to_install.name
>> )
>>
>> if not page: # Finally, look at the /simple/ index itself
>> page = request_url("https://pypi.python.org/simple/")
>>
>> # From here, look at the page to discover things.
>>
>>
>> As far as I can tell a lot of this is largely historical.
>>
>> The /simple/{name}/{verison}/ pages come from a time when there wasn't a simple
>> index I think and sometimes packages would need to go to /pypi/foo/version/
>> in order to actually get a list of things. However we now do have the simple
>> API and AFAICT the /simple/ API does not nor has ever had a reponse for
>> /simple/{name}/{version}/. This always 404's and falls back to the
>> /simple/{name}/. I would like to consider this URL unsupported in pip and remove
>> checking for it. It will reduce the number of needless HTTP requests by one
>> per pinned version.
>>
>> Does anyone know anything this will break?
>
> Sounds fine to me to remove the /{version}/ check. FWIW devpi also doesn't
> support it.
>
>> The other thing that happens is if the /simple/{name}/ thing 404's it'll
>> fallback to /simple/. This is done so that if someone mistypes a name in a way
>> that is still considered equivilant after normalization is applied, instead
>> of a 404 they get the /simple/ page and the tooling can discover the name from
>> there.
>>
>> If you remember back a little while ago I changed PyPI so that it considered
>> the normalized form of of the name the "cannonical" name for the simple index,
>> this means that tooling will be able to know ahead of time if a project called
>> say "Django" should be requested with /simple/Django/ or /simple/django/.
>>
>> What I would like to do now is remove the fallback to /simple/. If we fall back
>> to that it is a 2.1MB download that occurs which is a fairly big deal and can
>> slow down a ``pip install`` quite signifcantly. I have a PR against
>> bandersnatch which will make bandersnatch generate a /simple/{name}/ URL where
>> name is the normalized form, and another PR against pip which will cause it
>> to always request the normalized form. When both of these land it would mean
>> that the only time pip will fallback to /simple/ is:
>>
>> 1. If someone is using a *new* pip (with the PR merged) but with a mirror that
>> doesn't support the normalized form of the URLs (old bandersnatch,
>> pep381client, maybe others?)
>
> I think devpi-server would also still redirect on normalized form to
> the "real name" but we plan to change that to behave as pypi does now
> (serve everything under the normalized name). How/do you plan to detect
> if the "normalized form" is supported from a server?
Yea I forgot to explicitly mention devpi-server, but I knew it did the redirect so
it would be fine there.
I don’t plan on detecting it, I just plan on sending it through the normal
deprecation cycle in pip. Since this doesn’t reflect any changes on the PyPI
side beyond what's already been done, and the changes to bandersnatch/mirrors
can be made in an completely backwards compatible fashion the only problem
would be when you use a *new* pip with an older mirroring client or index
server. I think a 2 release deprecation cycle (so pip 6.0 is our next release,
it would be deprecated there and ultimately removed in pip 8.0) is ample time
for folks to see the warning and adjust their own internal mirrors.
>
>> 2. If someone typed ``pip install <foo>`` where <foo> is a thing that doesn't
>> actually exist.
>
> If we had a meta header "pypi-simple-protocol-version" set to to "1" by
> pypi.python.org and compliant servers we could avoid all falling back to
> simple/ safely. You do a single lookup for each project and are done,
> safely, no falling back to simple/ at all.
>
> FWIW devpi-server on simple/<name> already returns a "200" with an empty
> link set for projects that don't exist. I've never had complaints from
> users about this hack. Note that pip/easy_install don't behave much
> differently today if they get an empty link set on simple/<name> or a
> 404 and then no match on the full simple/ page.
Correct, if there’s no match on the /simple/ page, then it’s functionally equivalent
to getting a blank /simple/{name}/ except there’s one less HTTP request made.
I don’t think we need to introduce a header, other than the change to serve
normalized URLs by default this isn’t technically a change in what PyPI serves,
just a change in what the installers are expected to go and fetch.
>
>> Does anyone have any complaints if pip stopped falling back to /simple/ once
>> the bandersnatch PR is merged and released?
>
> I am fine, preferably in lieu with introducing indication from pypi.python.org
> about the simple-api version.
>
>> Further more does anyone have any problems with narrowing the "supported URLS"
>> of the "simple API" to /simple/ and /simple/<normalized-name>/ and make
>> fetching /simple/ optional?
>
> Thanks for cleaning up the protocols.
Heh, no worries :) Hopefully we’ll end up with this all simplified so that pip/index.py
can be cleaned up and simplified and a nice PEP can be written up with a final
“this is the Simple API” and no longer need to have it spread over multiple PEPs,
setuptools documentation, and pip/setuptools source code.
---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140918/fc007d6b/attachment-0001.html>
More information about the Distutils-SIG
mailing list