[Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting

[holger krekel]

On Tue, Oct 07, 2014 at 11:40 +0100, Paul Moore wrote:
> On 7 October 2014 11:09, holger krekel <holger at merlinux.eu> wrote:
> > Well, the main benefit of PEP438 was that it removed random crawling for
> > some 90% of the packages on the package index, speeding up and making
> > installs more reliable.  And it did that without breaking backward
> > compatibility.
> 
> The setuptools index page is 1.4MB in size. Most of that can be
> ignored, but it still has to be downloaded and parsed. Whether the
> data that setuptools includes in its long_description is useful is
> arguable, but irrelevant - the fact is that as things stand, it is
> there and it causes issues.
> 
> PEP 470 would result in all of the unneeded entries in the simple
> index for setuptools being removed, which avoids the need for client
> tools (and I'm not talking just about pip here, but also about one-off
> scripts, which is the sort of thing I write a lot) to trawl through
> all of that data. And it does so without the setuptools project having
> to change how it writes its PyPI page (i.e., the project
> long_description).
> Arguably, that's equally a way of avoiding breaking backward compatibility...
> 
> > The second could be done without breakage alltogether i think:  at one
> > time all external urls are auto-registered as external indexes
> > and they are presented on the simple page with some meta information
> > that does not confuse older pips/easy_installs.  Newer pips/easy_installs
> > can then provide nice error messages.  Older pips can continue to use
> > the PEP438 options.  And easy install can continue to work.
> 
> Setuptools has 255 internal links to files hosted on PyPI. And about
> 11,000 other links. (I just checked that 3 times, as I couldn't
> believe it, but it *seems* to be right :-(). Removing duplicates, 337
> unique links. Are you suggesting pip presents all of those as possible
> external indexes?

No, i effectively suggest that PyPI would present just 2 index links, 
those which currently are attributed as rel={download,homepage}.
Those two index links would be put into the new "extra indexes field"
on pypi with a note like "the following indexes were extracted from old
release data" which newer pip versions could present to the user.
For older pip/easy_installs things would just continue to work 
but they'd get a shorter setuptools simple page.

best,
holger

> I'm sure you can argue that setuptools has (badly!) misused the
> link-handling support in PyPI. And that it's a one-off special case.
> But how do we document to projects that they shouldn't do things like
> this? How do we even define what "things like this" are? Don't include
> links in your project description unless they are external indexes?
> 
> Paul.
>