[Distutils] PEP470 installation security problems

Wed Oct 8 16:04:59 CEST 2014

> On Oct 8, 2014, at 9:40 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
> On 08.10.2014 15:05, Donald Stufft wrote:
>> 
>>> On Oct 8, 2014, at 8:55 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>> 
>>> On 08.10.2014 14:30, Nick Coghlan wrote:
>>>> On 8 October 2014 22:22, Donald Stufft <donald at stufft.io> wrote:
>>>>> 
>>>>>> On Oct 8, 2014, at 8:17 AM, holger krekel <holger at merlinux.eu> wrote:
>>>>>> 
>>>>>> Also, i am worried on principle grounds if pip maintainers are putting
>>>>>> themselves outside PEP reach, yet pip is distributed along with Python.
>>>>> 
>>>>> We’re not “putting ourselves outside of PEP reach”. We are an external
>>>>> project and we are not bound by the PEP process. Devpi, py.test, Django,
>>>>> requests, etc are also not bound by the PEP process.
>>>> 
>>>> Note also that even for CPython itself, it is *up to us as core
>>>> developers* to decide when something needs to be escalated through the
>>>> PEP process. The vast majority of CPython changes are handled directly
>>>> through the issue tracker, and there's still the occasional change
>>>> that doesn't even make it that far (e.g. if we notice a problem while
>>>> working on something else, we have the option of just committing the
>>>> fix directly).
>>>> 
>>>> PEPs are primarily for changes which have broad ecosystem implications
>>>> where the additional overhead is justified. We don't write PEPs for
>>>> every change to the CPython command line interface (e.g. there's no
>>>> PEP for isolated mode), and the same kind of assessment of external
>>>> impact applies to pip and the PyPA in general when decided whether a
>>>> change can be handled within the scope of an individual project, or if
>>>> it needs to be escalated for broader discussion.
>>> 
>>> I don't follow Donald's reasoning and I'm not sure I understand
>>> whether your comments are meant as clarification of pip being
>>> subject to the PEP process or support for Donald's reasoning :-)
>>> 
>>> Changes to pip and PyPI *do* have a global effect on the Python
>>> ecosystem and thus need to be covered by the PEP process.
>>> 
>>> If pip decides to go with a strategy that ignores this, I think we
>>> have a problem. The core developers put trust into pip when allowing
>>> it to (effectively) get distributed with Python and making it the
>>> default Python packaging manager. Please use that trust with the
>>> appropriate care and respect.
>> 
>> I don’t think we’ve *ever* not used that trust with care and respect and
>> we’ve been trusted by the Python community for far longer than PEP 453
>> has existed. We attempt to follow PEPs where we can and where they make
>> good sense. Nobody on the pip team is saying we’re going to flat out
>> ignore PEPs or whatever.
>> 
>> We (or at least I am) are saying that dictating UX via PEP process has
>> been shown to us *not* to work and that we are not obligated to implement
>> or listen to a PEP. This was explicitly spelled out in PEP 453 that we
>> remain an external project even with the fact we’re now bundled with
>> Python. This does not mean we won’t generally try to use the PEP process
>> where our changes have cross cutting concerns between different projects
>> but it does mean that we implement or follow PEPs at our discretion. This
>> isn’t up for debate, it was an explicit inclusion in PEP 453 and if there
>> was a problem with pip maintaining it’s own project the time to bring that
>> up was a year ago. 
> 
> The intention of PEP 435 was to enable pip to evolve independent
> of the Python release process, which is a good thing.
> 
> However, your comment that "We are an external project and we are not
> bound by the PEP process." doesn't really pan out in the light of the PEP's
> requirement that "The maintainers of the bootstrapped software and the
> CPython core team will work together in order to address the needs of both."
> 
> If pip maintainers don't feel they are bound by PEPs, you could argue
> that you are also not bound by PEP 435, which would result in a
> rather pointless cooperation setup :-)
> 
> Note that I'm not trying to say that you are actually not respecting the
> PEP process. I'm just concerned about comments like the above causing
> unnecessary heat in discussions.

I feel like this whole “Is pip subject to PEPs” thing went way off the
rails somewhere. Originally it was just “A PEP can’t mandate to an installer”
which is true, pip is the only installer bundled with Python and I try
to write my PEPs to be installer agnostic. I think it also got bound up
in the fact that I/we feel pretty strongly that dictating a UX in a PEP
to pip doesn’t work (discovered through experience with PEP 438) and thus
we’re unlikely to listen to a PEP that dictates a UX that we feel is
bad again because it’s turned out to be bad idea for us (although it’s more
likely that such a PEP wouldn’t get accepted to begin with I think).

Somewhere that morphed to pip is not subject to PEPs, I think with the
suggestion that the changes Holger is asking to be made are specific to
the pip functionality and not to PEP 470 so he should raise those issues
on the pip tracker as they aren’t part of PEP 470 and that pip doesn’t
generally use the PEP process for our behavior). From there is snowballed
into this argument which I think is likely to just be circular and largely
pointless as I don’t think a situation where pip flat out refuses to follow
a reasonable PEP (or even a request from python-dev) is likely to come up
so the difference is academic.

> 
> I'd also like to request that you take Holger's concerns more
> seriously, perhaps add him as PEP author and let him participate
> in clarifying it (if he still feels like investing time in this).

I take all concerns and feedback seriously else I wouldn’t spend the many
hours I’ve spent just this morning responding to them. I don’t grok what
Holger’s actual concern is so it’s hard to turn those concerns into anything
actionable I can actually do on the PEP. I’ve asked for him (if he desires!)
to give an actual example of something he’d change in the PEP to see if maybe
that would make it clearer to me what he’s actually concerned about in
relation to the PEP. I’m going to remove the one half a sentence that mentions
a private repository in any capacity but I have a hard time believing that
using it as one example in a list is what Holger’s concern is so I feel like
that probably won’t fully address it.

> 
> PEPs are never perfect and there's always room for improvement.
> 
> Thanks,
> -- 
> Marc-Andre Lemburg
> eGenix.com
> 
> Professional Python Services directly from the Source
>>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
> 
> ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
> 
> 
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>           Registered at Amtsgericht Duesseldorf: HRB 46611
>               http://www.egenix.com/company/contact/

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA