<div dir="ltr">Ah, I didn't think of that- good point. +1 to your suggested approach.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Aug 28, 2014 at 3:41 PM, Donald Stufft <span dir="ltr"><<a href="mailto:donald@stufft.io" target="_blank">donald@stufft.io</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Since pip 1.4 it does yes, however the problem here is that typically bandersnatch</div>
<div>mirrors are simply hosted by plain static web servers and don’t require any sort of runtime logic.</div><div><div class="h5"><br><div><blockquote type="cite"><div>On Aug 28, 2014, at 6:39 PM, Joe Smith <<a href="mailto:yasumoto7@gmail.com" target="_blank">yasumoto7@gmail.com</a>> wrote:</div>
<br><div><div dir="ltr">Naive question- does pip send over a UserAgent (or something) that contains a version number the server can use to determine which behavior to default to?<div><br></div><div>That would allow a deprecation cycle of N months or so that will let people upgrade from 1.5 to 1.6. We could then watch usage of 1.5 decrease over time until it's a non-factor.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Aug 28, 2014 at 3:26 PM, Donald Stufft <span dir="ltr"><<a href="mailto:donald@stufft.io" target="_blank">donald@stufft.io</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div><br><div><blockquote type="cite"><div>On Aug 28, 2014, at 6:09 PM, Donald Stufft <<a href="mailto:donald@stufft.io" target="_blank">donald@stufft.io</a>> wrote:</div>
<br><div><div style="word-wrap:break-word"><br><div><blockquote type="cite"><div>On Aug 28, 2014, at 2:58 PM, Donald Stufft <<a href="mailto:donald@stufft.io" target="_blank">donald@stufft.io</a>> wrote:</div><br><div>
<div style="word-wrap:break-word"><div><div>Right now the “canonical” page for a particular project on PyPI is whatever the</div><div>author happened to name their package (e.g. Django). This requires PyPI to have</div><div>
some "smarts" so that it can redirect things like /simple/django/ to</div><div>/simple/Django/ otherwise someone doing ``pip install django`` would fall back</div><div>to a much worse behavior.</div><div><br></div>
<div>If this redirect doesn't happen, then pip will issue a request for just</div><div>/simple/ and look for a link that, when both sides are normalized, compares</div><div>equal to the name it's looking for. It will then follow the link, get</div>
<div>/simple/Django/ and everything works... Except it doesn't. The problem here</div><div>comes from the external link classification that we have now. Pip sees the</div><div>link to /simple/Django/ as an external link (because it lacks the required</div>
<div>rels) and the installation finally fails.</div><div><br></div><div>The /simple/ case rarely happens when installing from PyPI itself because of</div><div>the redirect, however it happens quite often when someone is attempting to</div>
<div>instal from a mirror instead. Even when everything works correctly the penality</div><div>for not knowing exactly what name to type in results in at least 1 extra http</div><div>request, one of which (/simple/) requires pulling down a 2.1MB file.</div>
<div><br></div><div>To fix this I'm going to modify PyPI so that it uses the normalized name in</div><div>the /simple/ URL and redirects everything else to the non-normalized name. I'm</div><div>also going to submit a PR to bandersnatch so that it will use normalized names</div>
<div>for it's directories and such as well. These two changes will make it so that</div><div>the client side will know ahead of time exactly what form the server expects</div><div>any given name to be in. This will allow a change in pip to happen which</div>
<div>will pre-normalize all names which will make the interaction with mirrors better</div><div>and will reduce the number of HTTP requests that a single ``pip install`` needs</div><div>to make.</div></div><br><div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div>---</div><div>Donald Stufft</div><div>PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA</div></div></div>
</div>
<br></div>_______________________________________________<br>Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org" target="_blank">Distutils-SIG@python.org</a><br><a href="https://mail.python.org/mailman/listinfo/distutils-sig" target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br>
</div></blockquote></div><div><br></div><div><div>Hm, so here’s the problem.</div><div><br></div><div>I have this implemented and deployed to TestPyPI, it works great!</div><div><br></div><div>However, the next step is to make the change to bandersnatch so that it saves</div>
<div>things using their normalized name instead of using their "proper" name. Doing</div><div>this will trigger it so that everyone using pip 1.5 won't be able to install</div><div>anything from that mirror unless it's name is specified as the normalized name</div>
<div>(e.g. ``pip install Django`` will fail without --allow-unverified but</div><div>``pip install django`` will work). This would be fixed with pip 1.6 (since</div><div>it would know to "normalize" the name before fetching the URL).</div>
<div><br></div><div>The same thing will occur if we make the change in pip first, it would</div><div>normalize names so you'd need to use --allow-unverified for everything because</div><div>it would act as if you typed ``pip install django`` instead of ``pip install</div>
<div>Django``.</div><div><br></div><div>To my knowledge, this *only* will affect pip 1.5.x.</div><div><br></div><div>So the only way forward I can see to make this change, which I think is a good</div><div>change and will remove a big "gotcha" from using a mirror, is to coordinate</div>
<div>a release of bandersnatch that coincides with pip 1.6, and tell people they</div><div>need to upgrade in lockstep.</div><div><br></div><div>Does anyone have any other ideas?</div></div><br><div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div>---</div><div>Donald Stufft</div><div>PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA</div></div></div>
</div>
<br></div>_______________________________________________<br>Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org" target="_blank">Distutils-SIG@python.org</a><br><a href="https://mail.python.org/mailman/listinfo/distutils-sig" target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br>
</div></blockquote></div><div><br></div></div></div><div><div>Just thought of this, if the normalized name doesn’t match the "real" name,</div><div>then add entries for both. This will make it so that pip 1.5 continues to work</div>
<div>and pip 1.6+.</div></div><div><br><div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div>---</div><div>Donald Stufft</div><div>PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA</div></div></div>
</div>
<br></div></div><br>_______________________________________________<br>
Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org" target="_blank">Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br>
<br></blockquote></div><br></div>
</div></blockquote></div><br><div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div>---</div><div>Donald Stufft</div><div>PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA</div></div></div>
</div>
<br></div></div></div></blockquote></div><br></div>