[Catalog-sig] V4 Pre-PEP: transition to release-file hosting on PYPI

Carl Meyer carl at oddbird.net
Sat Mar 16 00:16:19 CET 2013


tl;dr: I see your points, we'll change the PEP to allow clients to use
hostnames instead of the rel attributes if they prefer. More comments below:

On 03/15/2013 12:59 PM, PJ Eby wrote:
> That's the bit I don't like.  The security model is that if it's not
> allowed by allowed-hosts, it's *not allowed*.  Introducing a way to
> sneak something past allow-hosts is a bad idea, because it means
> people either have to explicitly widen their allow-hosts to arbitrary
> hosts, or else that you can't actually enforce an allowed-hosts
> policy, or that you need to learn a whole bunch of options to
> implement it.
> 
> ISTM that this is a bad design choice for users, and I'm not
> comfortable with this without some way to define the allowed
> "internal" hosts based in some way on the base index URL.  Not just
> for ease of automated translation, but so that *users* can know who
> they're dealing with, and easily predict the effects of their chosen
> options.
> 
> A frequent refrain has been, "users don't know they're downloading
> stuff from places other than PyPI", so if this new approach allows
> downloads from somewhere other than *.pypi.python.org when you've
> chosen pypi.python.org as your index, ISTM the proposal is failing to
> meet its original goals.  As the PEP is written, PyPI could change out
> to a different CDN each week or use different ones for different
> files, and users would be back in the position of not being sure where
> stuff is coming from.

I guess the key question is the definition of "places other than PyPI."
I think a CDN that is part of the index's architecture is just as much
"part of PyPI" whether it's on the same domain or not. But I understand
the difficulty integrating this with the --allow-hosts option in a way
that maintains a clear and simple UI.

> I'm fine with extending the default host matching to
> "indexhost,*.indexhost" if we want to leave more of an option for PyPI
> and other indexes to use a CDN.  But I'm not sure how much point to it
> there is, since a /simple index is static, and small in size compared
> to the downloads, so you might as well host a copy of the /simple
> index alongside the downloads, and make the index pypicdn.com/simple
> or whatever in the first place.  (In other words, not a lot of benefit
> to splitting a static index from its associated files, so why support
> it?)

Putting the /simple/ API on a CDN isn't quite that easy because it
currently involves some server-side redirects to effectively make
project names case-insensitive. I think in a hypothetical
re-architecture of PyPI there may be good security reasons to put
user-uploaded files on a different domain from dynamic portions of the
API (Donald alluded to this, more discussion at
http://security.stackexchange.com/questions/11756/is-it-safe-to-serve-any-user-uploaded-file-under-only-white-listed-mime-content).

So I think this issue may come up again in the future. But I'm fine with
deferring it in this PEP for now...

>> PyPI wouldn't be enforcing a UI on you here, just providing metadata
>> that you can use as you wish.
> 
> That's not what the PEP says.  It does in fact *mandate* the use of
> the rel attributes.  So if somebody adds an "external link" that
> actually points back to PyPI, technically I'm not supposed to use it
> unless it's been explicitly authorized.  ;-)
> 
> I'd really prefer to see explicit language that says the rel
> information is advisory only and that installers aren't required to
> parse it, let alone use it.  At the moment, the PEP is a substantial
> departure from the version I agreed with.

Ok, pending agreement from Holger I'll make a change in the PEP to
explicitly allow clients to make decisions based on either the rel
attributes or based on hostnames. Would that be sufficient to address
your concerns?

Carl

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20130315/17d9fbc3/attachment.pgp>


More information about the Catalog-SIG mailing list