[Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files

Donald Stufft donald at stufft.io
Wed Mar 13 20:46:37 CET 2013

On Mar 13, 2013, at 3:33 PM, "M.-A. Lemburg" <mal at egenix.com> wrote:

> On 13.03.2013 20:08, Donald Stufft wrote:
>> On Mar 13, 2013, at 2:57 PM, "M.-A. Lemburg" <mal at egenix.com> wrote:
>>> On 13.03.2013 12:21, holger krekel wrote:
>>>> [V3 proposal]
>>> I must say, don't like this change in motivation compared
>>> to V1 and V2.
>>> The original of the discussion was to make PyPI more secure
>>> and the installation process faster and more reliable
>>> by moving away from crawling arbitrary external web pages.
>>> Both can be had by:
>>> * limiting the crawling to package author defined specific
>>> URLs, with added hashes to make sure that the URLs and
>>> their target content is not modified (this is the securing
>>> external downloads part - see here for an example:
>>> https://pypi.python.org/pypi/egenix-pyopenssl/,
>>> and
>>> * adding a way for the package authors to say "PyPI, please go
>>> ahead and cache/copy my distributions files" (this is the
>>> increase download reliability part - can be had by doing
>>> opt-in CDN caching/proxying of external links via PyPI)
>>> Now, with V3 of the proposal, you are moving towards a system
>>> that basically says "do it this way, or stay out of our eco
>>> system", which, in my book, is not what the Python eco system
>>> is all about.
>> I don't see how? The -with-externals index will still contain all the existing links, and indeed PJ Elby has already stated that setuptools will move to support this index by default but with proper warnings to people so they know they are installing a package off site.
>> This allows existing tools to be moved to a secure by default position. Allows future tools to choose if they want to enable the existing behavior through use of -with-externals (hopefully with a warning or opt-in sort of thing like laid out by PJE, but it's certainly not required). And even allows users of existing tools to opt into the old behavior via the -i option.
>> Maybe i'm missing it but in what way does this force authors to "do it this way or stay out of our eco system" since all the same options are available as there are today?
> The proposal marks all external links as evil, and instead of
> making external links more secure, the user is left with the option
> to either not enable external links at all, or to let the
> "devil" in :-)

It doesn't mark them as evil, it marks them as requiring users to opt into them. Authors are free to not publish their packages directly to PyPI and users are free to opt in to installing the external urls that the authors haven chosen to publish. Further more it gives package authors complete control over what urls appear on their simple index page.

ISTM that this is even friendlier than before because now both sides have explicitly decided to use those urls, instead of it being completely implicit on one said, and partially implicit on the other.

> That's not nice. It's also security theater.

It's not security theater, it moves the defaults to more secure. Further work can (and will be) to ensure that for those users and authors who opt into the external urls it's still secure while again requiring both sides to explicitly opt into it.

> The real problem is unreviewed code getting executed by users,
> or worse, automated build systems. Yet, we let users believe
> that everything is secured on PyPI.

"We"? I' don't think anyones ever said that *everything is secured on pypi*. The best the PyPI infrastructure and tooling can do (security wise) is to try and make as sure as possible then when you ask for foo==X.Y PyPI currently can't make that claim for external links.

On top of that many users (and i'd wager most users) are not aware that when they install something it reaches outwardly to other hosts. This proposal makes it so they *are* aware so they opt into potentially lowering their downtime and they opt into exposing details to external hosts (which may or may not be SSL secured).

> Taking an extreme position, it would probably be better just
> leave everything as it is and instead educate users about the
> risk they are taking with a "pip install AngryBirds", signed
> with keys issued by the PSF on the official PyPI server,
> delivered straight to your drive via the latest in crypto
> technology, only to wipe your notebook...
> But then, I don't like extreme positions, so would rather
> like to incrementally improve the situation both from the
> server and the client side, both addressing user and author
> concerns, and keeping the Python eco system a friendly place
> to be.
>>> Your V2 was much more inviting in this respect.

This gives _all_ the abilities of the current system (besides spidering random urls) with *more* control given to the authors as to what exists on their various index pages. This is a net win for everyone involved. The only "loss" is that projects that choose to host externally to PyPI will have people trying to install it told to explicitly allow it (as mentioned by PJ Elby).

> -- 
> Marc-Andre Lemburg
> eGenix.com
> Professional Python Services directly from the Source  (#1, Mar 13 2013)
>>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>           Registered at Amtsgericht Duesseldorf: HRB 46611
>               http://www.egenix.com/company/contact/

Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20130313/d5ad5581/attachment-0001.pgp>

More information about the Catalog-SIG mailing list