[Distutils] PEP470 installation security problems

[Donald Stufft]

> On Oct 8, 2014, at 2:35 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> On 08.10.2014 16:04, Donald Stufft wrote:
>> 
>>> I'd also like to request that you take Holger's concerns more
>>> seriously, perhaps add him as PEP author and let him participate
>>> in clarifying it (if he still feels like investing time in this).
>> 
>> I take all concerns and feedback seriously else I wouldn’t spend the many
>> hours I’ve spent just this morning responding to them. I don’t grok what
>> Holger’s actual concern is so it’s hard to turn those concerns into anything
>> actionable I can actually do on the PEP.
> 
> Holger has made his points very clear in his emails.
> 
> If you don't follow/grok his reasoning it may indeed be better to
> have him edit the PEP to add his improvements/changes.
> 
> I share his view that it is not necessary to break existing
> setups to add multi-index support. This can be implemented as
> simple extension to what we already have:
> 
> """
> Simply add the possibility for authors to register external indexes,
> have pip, setuptools, et al. crawl these in addition to what's
> up on the PyPI package page (using the logic that has existed in
> these tools for years) and then let the author decide whether they
> want to remove existing downloads from PyPI or not.
> 
> This allows for older installations to continue working, while
> also (optionally) supporting a setup which does not use PyPI for
> hosting at all.
> “""

His backwards compatibility point I understood completely and I responded that
I believe that maintaining backwards compatibility for a minority of projects,
where that backwards compatibility is almost entirely unsafe, is not more
important than making ``pip install`` safe and it is actively preventing us
making the pip repository code better. The PEP reflects that view, Holger and
you may not agree with it and that's fine, I'm not going to compromise that
with maintaining compat for a tiny fraction of people/projects.

The thing I don't understand is Holger's worry about using the multi repository
support for private projects. The PEP is entirely about using an existing feature
for public projects. Private projects are mentioned a grand total of once in
the entire PEP and that's just saying that the ability to specify other
repositories is also useful for... and then lists a couple items, one of which
is internal company repositories. So I'm not sure what I'm supposed to do with
Holger's concern about --extra-index-url which don't apply to the PEP at all as
far as I can tell which is where I'm looking for some clarification. How does
Holger want me to address the use of this feature in a mostly unrelated to the
PEP fashion in this PEP?

> 
> BTW: For eGenix we've chosen to use a different approach, one
> that is based on a Python web installer. I gave a talk about this at
> PyCon UK, in case you're interested:
> https://downloads.egenix.com/python/PyCon-UK-2014-Python-Web-Installer-Talk.pdf
> (talk video here:
> http://www.egenix.com/library/presentations/PyCon-UK-2014-Python-Web-Installer/)
> This solves the issues with the pip user experience for our packages,
> solves the download selection issues for the binaries, works with
> all Python versions we support and assures that the downloads
> are safe. It's still work in progress, but already quite usable.
> 

I’ve been pointed at your web installer and poked at it a little bit. I
don’t have any specific points about it since I only really skimmed it,
hopefully you’re doing all that needs to be done to secure the downloads,
but otherwise I’m glad you found something that gives you more control
over the process and that still works well with the newer policies of
things.

Maybe it’d make sense to also explicitly mention this as an additional
option? Is there a tool you’re using to manage all this or is it all
one-off and specific to eGenix?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA