[Distutils] Handling Case/Normalization Differences

[Joe Smith]

Ah, I didn't think of that- good point. +1 to your suggested approach.


On Thu, Aug 28, 2014 at 3:41 PM, Donald Stufft <donald at stufft.io> wrote:

> Since pip 1.4 it does yes, however the problem here is that typically
> bandersnatch
> mirrors are simply hosted by plain static web servers and don’t require
> any sort of runtime logic.
>
> On Aug 28, 2014, at 6:39 PM, Joe Smith <yasumoto7 at gmail.com> wrote:
>
> Naive question- does pip send over a UserAgent (or something) that
> contains a version number the server can use to determine which behavior to
> default to?
>
> That would allow a deprecation cycle of N months or so that will let
> people upgrade from 1.5 to 1.6. We could then watch usage of 1.5 decrease
> over time until it's a non-factor.
>
>
> On Thu, Aug 28, 2014 at 3:26 PM, Donald Stufft <donald at stufft.io> wrote:
>
>>
>> On Aug 28, 2014, at 6:09 PM, Donald Stufft <donald at stufft.io> wrote:
>>
>>
>> On Aug 28, 2014, at 2:58 PM, Donald Stufft <donald at stufft.io> wrote:
>>
>> Right now the “canonical” page for a particular project on PyPI is
>> whatever the
>> author happened to name their package (e.g. Django). This requires PyPI
>> to have
>> some "smarts" so that it can redirect things like /simple/django/ to
>> /simple/Django/ otherwise someone doing ``pip install django`` would fall
>> back
>> to a much worse behavior.
>>
>> If this redirect doesn't happen, then pip will issue a request for just
>> /simple/ and look for a link that, when both sides are normalized,
>> compares
>> equal to the name it's looking for. It will then follow the link, get
>> /simple/Django/ and everything works... Except it doesn't. The problem
>> here
>> comes from the external link classification that we have now. Pip sees the
>> link to /simple/Django/ as an external link (because it lacks the required
>> rels) and the installation finally fails.
>>
>> The /simple/ case rarely happens when installing from PyPI itself because
>> of
>> the redirect, however it happens quite often when someone is attempting to
>> instal from a mirror instead. Even when everything works correctly the
>> penality
>> for not knowing exactly what name to type in results in at least 1 extra
>> http
>> request, one of which (/simple/) requires pulling down a 2.1MB file.
>>
>> To fix this I'm going to modify PyPI so that it uses the normalized name
>> in
>> the /simple/ URL and redirects everything else to the non-normalized
>> name. I'm
>> also going to submit a PR to bandersnatch so that it will use normalized
>> names
>> for it's directories and such as well. These two changes will make it so
>> that
>> the client side will know ahead of time exactly what form the server
>> expects
>> any given name to be in. This will allow a change in pip to happen which
>> will pre-normalize all names which will make the interaction with mirrors
>> better
>> and will reduce the number of HTTP requests that a single ``pip install``
>> needs
>> to make.
>>
>>  ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>> Hm, so here’s the problem.
>>
>> I have this implemented and deployed to TestPyPI, it works great!
>>
>> However, the next step is to make the change to bandersnatch so that it
>> saves
>> things using their normalized name instead of using their "proper" name.
>> Doing
>> this will trigger it so that everyone using pip 1.5 won't be able to
>> install
>> anything from that mirror unless it's name is specified as the normalized
>> name
>> (e.g. ``pip install Django`` will fail without --allow-unverified but
>> ``pip install django`` will work). This would be fixed with pip 1.6 (since
>> it would know to "normalize" the name before fetching the URL).
>>
>> The same thing will occur if we make the change in pip first, it would
>> normalize names so you'd need to use --allow-unverified for everything
>> because
>> it would act as if you typed ``pip install django`` instead of ``pip
>> install
>> Django``.
>>
>> To my knowledge, this *only* will affect pip 1.5.x.
>>
>> So the only way forward I can see to make this change, which I think is a
>> good
>> change and will remove a big "gotcha" from using a mirror, is to
>> coordinate
>> a release of bandersnatch that coincides with pip 1.6, and tell people
>> they
>> need to upgrade in lockstep.
>>
>> Does anyone have any other ideas?
>>
>>  ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>> Just thought of this, if the normalized name doesn’t match the "real"
>> name,
>> then add entries for both. This will make it so that pip 1.5 continues to
>> work
>> and pip 1.6+.
>>
>>  ---
>> Donald Stufft
>> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>>
>>
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
>  ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140828/92ec178c/attachment.html>