[Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

[Carl Meyer]

On 03/12/2013 03:22 PM, PJ Eby wrote:
> Here's a story to illustrate what I mean:
> 
> Joe wants to install foo.  He runs "easy_install Foo".  Foo is hosted
> externally to PyPI, so easy_install says:
> 
> URL foo.com/downloads/foo-1.2.tgz BLOCKED by allow-hosts option --
> install failed.
> 
> (Or words to that effect; I'd have to check the source to get you the
> exact phrasing).
> 
> The point is, Joe now *knows where to get foo from*, because PyPI
> still had the information.  Joe can now decide whether he wants to
> download it manually and inspect it first, expand his allow-hosts
> option, or give Foo a pass.
> 
> The proposals that call for banning all links from the /simple index,
> prevent Joe from being able to do this at all.

Ah, thank you! Yes, I was indeed missing that mode of getting the
information to the user. Makes perfect sense now.

> I would support making references to external links explicit rather
> than implicit.

Excellent.

>> I think the current "automatically suck in links to simple/" behavior is
>> only useful as a backwards-compatibility hack, which is why I think an
>> explicit switch to disable it (on by default for newly-registered
>> projects, slowly, gently, carefully migrated to on for existing
>> projects) is better than keeping this link-scraping behavior
>> indefinitely for all projects and asking package owners to clean up
>> their long-descriptions.
> 
> I would agree with dropping link parsing from the description field,
> provided that an alternative way is provided for projects to
> explicitly add external links to /simple, concurrent with the other
> changes.

So the other change I proposed is that these new explicitly-nominated
external links would not be added to the main simple/ index page for a
project, but to a with-external-links/ sub-page that includes all links,
internal and external. (This being, of course, subject to the same
package-owner-controlled migration process, nothing done abruptly). The
long-term benefits I see to making this tweak:

1) Users still using today's easy_install on RHEL in five years will
automatically get the benefit of safe-by-default (as each package owner
makes their migration) without needing to upgrade their easy_install.

2) Implementors of future installers can make explicit choices about
which set of links to ask for, without every single installer needing to
reimplement possibly-error-prone and possibly-subject-to-attack
host-comparison code.

I realize that this requires updating easy_install/pip/buildout in order
to take advantage of externally-hosted files in the new system, but
since end-user tooling updates are part of the plan either way, I think
in the spirit of safe-by-default it's preferable to require end-user
tooling updates to get access to less-safe options, rather than require
end-user tooling updates in order to become safer by default.

What do you think?

> Thank you for taking the time to engage and re-engage on this issue,
> and to "Explain It Like I'm Five" for me, with an illustration of an
> actual problematic use case.  ;-)

Of course, and likewise; I've learned a lot from this exchange and
appreciate you sticking with it and explaining things the second and
third time until I got it. :-)

Carl