[Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting

[Donald Stufft]

> On Oct 7, 2014, at 7:03 AM, holger krekel <holger at merlinux.eu> wrote:
> 
> On Tue, Oct 07, 2014 at 11:40 +0100, Paul Moore wrote:
>> On 7 October 2014 11:09, holger krekel <holger at merlinux.eu> wrote:
>>> Well, the main benefit of PEP438 was that it removed random crawling for
>>> some 90% of the packages on the package index, speeding up and making
>>> installs more reliable.  And it did that without breaking backward
>>> compatibility.
>> 
>> The setuptools index page is 1.4MB in size. Most of that can be
>> ignored, but it still has to be downloaded and parsed. Whether the
>> data that setuptools includes in its long_description is useful is
>> arguable, but irrelevant - the fact is that as things stand, it is
>> there and it causes issues.
>> 
>> PEP 470 would result in all of the unneeded entries in the simple
>> index for setuptools being removed, which avoids the need for client
>> tools (and I'm not talking just about pip here, but also about one-off
>> scripts, which is the sort of thing I write a lot) to trawl through
>> all of that data. And it does so without the setuptools project having
>> to change how it writes its PyPI page (i.e., the project
>> long_description).
>> Arguably, that's equally a way of avoiding breaking backward compatibility...
>> 
>>> The second could be done without breakage alltogether i think:  at one
>>> time all external urls are auto-registered as external indexes
>>> and they are presented on the simple page with some meta information
>>> that does not confuse older pips/easy_installs.  Newer pips/easy_installs
>>> can then provide nice error messages.  Older pips can continue to use
>>> the PEP438 options.  And easy install can continue to work.
>> 
>> Setuptools has 255 internal links to files hosted on PyPI. And about
>> 11,000 other links. (I just checked that 3 times, as I couldn't
>> believe it, but it *seems* to be right :-(). Removing duplicates, 337
>> unique links. Are you suggesting pip presents all of those as possible
>> external indexes?
> 
> No, i effectively suggest that PyPI would present just 2 index links, 
> those which currently are attributed as rel={download,homepage}.
> Those two index links would be put into the new "extra indexes field"
> on pypi with a note like "the following indexes were extracted from old
> release data" which newer pip versions could present to the user.
> For older pip/easy_installs things would just continue to work 
> but they'd get a shorter setuptools simple page.

I am not opposed to moving the rel={download,homepage} automatically from
link to metatag, I am opposed to leaving them in place in an attempt to
prioritize backwards compatibility over safety. The only thing I have
against automatically translating the old links to the new external hosting
metadata is that it’s going to be a lot of noise for authors who won’t know
which links are the correct links to use.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA