[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[Donald Stufft]

On May 11, 2014, at 4:50 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 11 May 2014 17:58, "Paul Moore" <p.f.moore at gmail.com> wrote:
> >
> > On 11 May 2014 08:38, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > > This confusion can likely be resolved by giving the obvious "allow external"
> > > name to the behaviour most users will want, and a more obscure name like
> > > "allow verifiable external" to the specialised behaviour folks like Stefan &
> > > MAL rely on.
> >
> > I'm struggling to reconcile Donald's assertion (based, I believe, on
> > his data from PyPI) that there are only 25 or so packages on PyPI that
> > are external but safe, and he's hot familiar with any of them, against
> > the comment that Stefan and MAL are affected by this change.
> 
> Let me be clear: this is *not* a technical decision from my perspective. It is a relationship management one, specifically in regards to maintaining the still fragile delegation of authority from python-dev to PyPA.
> 
> We currently have two core developers (Stefan Krah & Marc-Andre Lemburg) that are *very* unhappy with the way pip is evolving, because they favour the use of external hosting over uploading their packages to PyPI. While that is a minority opinion in the Python community at large, it still represents a significant proportion of the core developers that actually pay much attention to packaging issues. Regardless of the technical merits of PEP 438, that disagreement places a strain on the trust relationship between python-dev & PyPA, the same relationship we rely on as part of getting proposals like PEP 453 (and hopefully the eventual inclusion of ensurepip in a 2.7 maintenance release) approved.
> 
> 

Since we’re being clear, Marc-Andre Lemburg is not currently utilizing this feature. The egenix downloads are unverifiable and afaik have always been so. He has a hash that links to a page that links to some urls with hashes. If I recall correctly he tried to propose this a year ago and it was rejected in favor of what was codified in PEP438 at the time. I don’t know why he’s not doing it correctly, it could be he’s confused about how it works (which would be further proof that the option is confusing) or he just doesn’t care (in which case I’m not sure why he’s bothering to argue against it) or some other reason that I can’t currently think of.

Stefan was using it but has since removed it because he was upset over a *warning*. 
> Donald's proposal is to take a situation that Stefan and MAL are *already* unhappy with and make it even *worse*, by making it impossible to opt in to verifiable external links without also opting in to unverifiable ones.
> 
It's also making a situation that *a lot* of other people who use pip are unhappy with, and making it one they are happier with. My responsibility is to them, not to two developers, one of whom hasn’t ever used the feature (as far as I can tell) and the other of whom found a warning so untenable that he purposely broke his package for the users of pip/easy_install.
> Even with the PyPI numbers to back it up, the fast time line currently makes it possible to view that proposal as a fit of pique directed at Stefan & MAL, rather than as a well considered design decision.
> 
There’s a reason I didn’t just merge the PR. There’s a reason I pointed it here instead of leaving it to discuss amongst only the pip core team. I wanted people to be able to comment and make their views heard before it was done.
> By contrast, keeping the current "allow verifiable external links" behaviour available as a renamed option prevents that misreading of the situation: moving the problematic feature aside rather than deleting it entirely makes it much clearer that the purpose really is the officially stated one (making things less confusing for most users), and the timing is largely coincidental, with the python-dev discussion simply acting as a trigger for people to start seriously discussing ways to improve the usability of these options.
> 

The more options we have, the more end user confusion we get. Combining the existing options and then adding a third option makes that situation worse. I seriously don’t think people realize how much feedback I personally get about these things. I don’t *want* to have to apologize to people every single day because this shit is confusing as hell to them but I currently do. I can’t view that proposal as anything else but adding more shit I have to explain to them and the immediately apologize that it is so damn confusing. This is something I explain to people almost every single day, sometimes multiple times a day, trying to keep these options around is *hurting* everyone else. It’s something i’ve been thinking about what to do for ~2 months or so but I just hadn’t had the time to crunch the numbers to figure out what a reasonable course of action is.

You’re worried that this change is a (or will at least be perceived as such) FU to Stefan and MAL, I’m worried that not fixing this is a FU to *everyone else*.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140511/b8d53286/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140511/b8d53286/attachment.sig>