[Catalog-sig] V4 Pre-PEP: transition to release-file hosting on PYPI

[PJ Eby]

On Fri, Mar 15, 2013 at 1:39 PM, Carl Meyer <carl at oddbird.net> wrote:
> up to you whether you also want to use rel="internal" as a hint for
> implicitly (perhaps with warning) adding to --allow-hosts,

That's the bit I don't like.  The security model is that if it's not
allowed by allowed-hosts, it's *not allowed*.  Introducing a way to
sneak something past allow-hosts is a bad idea, because it means
people either have to explicitly widen their allow-hosts to arbitrary
hosts, or else that you can't actually enforce an allowed-hosts
policy, or that you need to learn a whole bunch of options to
implement it.

ISTM that this is a bad design choice for users, and I'm not
comfortable with this without some way to define the allowed
"internal" hosts based in some way on the base index URL.  Not just
for ease of automated translation, but so that *users* can know who
they're dealing with, and easily predict the effects of their chosen
options.

A frequent refrain has been, "users don't know they're downloading
stuff from places other than PyPI", so if this new approach allows
downloads from somewhere other than *.pypi.python.org when you've
chosen pypi.python.org as your index, ISTM the proposal is failing to
meet its original goals.  As the PEP is written, PyPI could change out
to a different CDN each week or use different ones for different
files, and users would be back in the position of not being sure where
stuff is coming from.

I'm fine with extending the default host matching to
"indexhost,*.indexhost" if we want to leave more of an option for PyPI
and other indexes to use a CDN.  But I'm not sure how much point to it
there is, since a /simple index is static, and small in size compared
to the downloads, so you might as well host a copy of the /simple
index alongside the downloads, and make the index pypicdn.com/simple
or whatever in the first place.  (In other words, not a lot of benefit
to splitting a static index from its associated files, so why support
it?)


> PyPI wouldn't be enforcing a UI on you here, just providing metadata
> that you can use as you wish.

That's not what the PEP says.  It does in fact *mandate* the use of
the rel attributes.  So if somebody adds an "external link" that
actually points back to PyPI, technically I'm not supposed to use it
unless it's been explicitly authorized.  ;-)

I'd really prefer to see explicit language that says the rel
information is advisory only and that installers aren't required to
parse it, let alone use it.  At the moment, the PEP is a substantial
departure from the version I agreed with.

(If there were to be any meaningful distinction in the links
themselves, I would think it'd more be whether, e.g. hash information
is available for the download.  That's a potentially relevant
distinction right now, in that PyPI automatically provides #md5 info.
Even so, I'm not sure that's enough of a distinction for anyone to
care about.)