[Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

Sat Feb 1 17:19:07 CET 2014

On Feb 1, 2014, at 10:45 AM, Donald Stufft <donald at stufft.io> wrote:

> It predates my involvement with pip, but pip used to have the feature where it
> did not require being installed into the virtualenv. It had the -E path/to/venv
> flag, and also had a PIP_RESPECT_VIRTUALENV env var that would have it
> automatically install into the activated virtualenv.
> 
> This was removed in the 1.1 release which was released on 2012-02-16. Again
> this predates my involvement with pip, however the changelog states that::
> 
>    * Removed ``-E``/``--environment`` option and ``PIP_RESPECT_VIRTUALENV``;
>      both use a restart-in-venv mechanism that's broken, and neither one is
>      useful since every virtualenv now has pip inside it. Replace ``pip -E
>      path/to/venv install Foo`` with ``virtualenv path/to/venv &&
>      path/to/venv/pip install Foo``.
> 
> It was removed in this commit: https://github.com/pypa/pip/commit/87dd157.
> 
> So perhaps it isn't that pip is lacking a useful feature, it's that pip tried
> it, as you did with distil, and due to pip's much larger user base it got
> a much more thorough real world experience with it and it turned out to be
> ultimately more trouble than it was worth? It could also have been a problem
> with how pip wrote it of course, and perhaps you've discovered a better way
> of writing it. However at the time it's obvious that the pain it was causing
> was enough to cause the pip developers to break backwards compatibility to
> remove that feature.
> 
> Why exactly was it removed? Well I can't find the reasoning beyond that commit
> so we'd need to get Carl or perhaps Jannis to come and tell us if they still
> remember. However it's clearly not that pip didn't just try or think of this,
> it did and it ultimately removed it.
> 
> On Feb 1, 2014, at 9:04 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
> 
>> On Sat, 1/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> 
>>> I have no idea how most of the internal design decisions on
>>> pip were made (and I neither need nor particularly want to
>>> know - that would be rather missing the point of collaborative
>>> development). However, you're trying to make out that the
>>> distil approach is so clearly superior that you don't
>> 
>> Not "superior" in terms of "clever". I've made a point of saying
>> that distil *doesn't do anything clever* in this area. If you're
>> saying "a bunch of design work", making "a bunch" sound like
>> "a lot", without any detailed knowledge of the design internals,
>> then how do you know if it was a hard engineering issue or a
>> trivial one? If I say "I didn't find it hard, and I'm not claiming to
>> be especially clever" then at least I'm speaking from a
>> knowledge of what it takes to achieve something, rather than
>> perhaps taking someone else's word for it. Decisions should
>> be properly informed, in my view.
>> 
>> Oh, and collaborative development sometimes involves
>> challenging prior work and assumptions. That's one way
>> in which things get better - we build on the past, but
>> don't remain enslaved to it.
>> 
>>> understand why anyone would ever have implemented it
>>> differently, and I'm trying to point out that there's an entirely
>>> plausible answer to that question that doesn't involve
>>> assuming poor engineering trade-offs on the part of the pip
>>> development team.
>> 
>> I didn't (at least, not intentionally) say anything about poor
>> engineering trade-offs made by anyone - what would be the
>> point of that? I do say that better engineering approaches might
>> exist, and suggest that the way distil does it one such approach.
>> If you can suggest technical flaws in the approach distil takes,
>> that's one thing, and I will be keen to look at any points you make.
>> If you have no time to examine distil and make pronouncements,
>> that's fine too - we're all busy people. But defending the status
>> quo seemingly on nebulous technical grounds like "a bunch", while
>> avowing no knowledge of the technical details, seems wrong.
>> How many milli-hodges are there in a bunch? ;-)
>> 
>>> No, things we do in our free time are the *best* opportunities
>>> to investigate things like that, because they're not necessarily
>>> goal oriented - they're more about satisfying our curiousity.
>> 
>> I don't disagree - my clumsily made point  was to to say
>> "volunteer" in the sense of "part-time", and therefore to
>> strengthen the fact that I well understand "limited resources".
>> 
>>> Consider the following two engineering design options:
>> [details snipped]
>>> Now *a priori*, you don't know how complex B is going to be.
>>> On the other hand, you already *know* A will work, because you
>>> already have working virtual environments.
>> 
>> That is fine as an *a priori* position, and it's the position I was in
>> when contemplating the development of distil. But we don't stay 
>> stuck in *a priori* forever: as I developed distil, it became clear
>> that alternative approaches can work. Of course there were
>> uncertainties at the beginning, but by doing the development
>> work I was able to understand better the nature of those, and
>> I could convert them into certainties of what would and wouldn't
>> work.
>> 
>>> And from a design uncertainty point of view, A also has the
>>> advantage: there is *no* design uncertainty, because you're
>>> just using the already-known-to-work virtual environment
>>> mechanism.
>> 
>> So you're saying PEP 405 was a mistake? We should just have
>> stuck with the tried-and-tested virtualenv, which required changing
>> every time a new Python version came out, because it used (of
>> necessity) what might be uncharitable to call "hacks"?
>> 
>>> Now, you *have* had time to explore option B in distil, and
>>> have discovered that it actually wasn't that hard after all.
>>> Cool, that's great and would be a nice feature to have in pip
>>> as well. However, the fact that *having already done it*, you
>>> discovered it wasn't all that difficult, doesn't change the fact
>>> that the level of effort involved was still greater than the
>>> nonexistent amount of work needed to get the existing solution
>>> working
>> 
>> Yes, but it's not as if I'm just presenting this aspect of distil *now*.
>> I certainly highlighted the salient features on this list *before* the
>> PEP 453 ship set sail and distil is pretty well documented on
>> readthedocs.org.
>> 
>>> and I noticed that you cut the part of the email pointing out the
>>> concrete technical benefits of being able to isolate the installer
>>> in addition to the installed components.
>> 
>> I wasn't trying to gloss over anything. If an installer maintains
>> backward compatibility, then isolating the installer is not
>> especially valuable. Even if the installer doesn't get it right,
>> using different versions of the installer is perfectly possible
>> (and easy with distil, as it's a single-file executable), and in
>> my experience it's not been unheard of to have had to
>> "downgrade" virtualenv or pip because a new release broke
>> something.
>> 
>>> It's one thing to disagree with a design decision after we
>>> have made the absolutely best possible case that we can
>>> for why that decision was the right choice, and still failing
>>> to convince ourselves. It's something else entirely to
>>> dismiss someone else's use case as invalid and *then*
>>> argue that an arguably simpler design (that doesn't handle
>>> the dismissed use case and was still only hypothetical at
>>> the time the decision was made)
>> 
>> Is this some revisionist view of history? I'm confused :-) The
>> original announcement I made about distil was on 26 March,
>> 2013:
>> 
>> http://thread.gmane.org/gmane.comp.python.distutils.devel/17336
>> 
>> The relevant features were fully documented in the version 0.1.0
>> documentation for distil, which was tagged on 22 March, 2013:
>> 
>> https://bitbucket.org/vinay.sajip/docs-distil/commits/04ac4badcab941e12ff6ecc0a3b6784fdc977e9b
>> 
>> This was on readthedocs.org and pythonhosted.org at the time
>> of the announcement.
>> 
>> The date on PEP 453 is 10 August, 2013 - at least four months
>> later:
>> 
>> http://www.python.org/dev/peps/pep-0453/
>> 
>> So, if no one who participated in the PEP 453 decision had time
>> to look at distil in that four month period, fair enough. It doesn't
>> mean that I'm somehow trying to bring something *new* to the
>> discussion *now*. I certainly made my reservations about the pip-
>> bundling route at the time it was being discussed here, but if
>> nobody was willing to look at alternatives, it's not something I
>> could help.
>> 
>> Of course, once the decision is made, what can you do but
>> defend it? I understand your position, and I'm not trying to
>> change anything. If anyone wants to provide specific, technical
>> feedback about any problems with distil's approach in any
>> area, I'll welcome that feedback - but I've already expressed my
>> view on non-technical battles.
>> 
>> Regards,
>> 
>> Vinay Sajip
>> _______________________________________________
>> Distutils-SIG maillist  -  Distutils-SIG at python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
> 
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig

Additionally I'd point out that, looking at how you've implemented it, it seems
to depend on the fact that distil is a single file and thus doesn't need
installed into site-packages. Pip however is not a single file and to use the
same technique as you did would require installing pip into each virtualenv or
doing some gross hacks to add pip to the sys.path dynamically.

Now it's possible that pip *could* be modified to fully support running as a
single file, however that would also entail getting our dependencies to support
that too. Some of them may work currently by accident but I would be
uncomfortable relying on support by accident and would want to have them
committed to running like that.

Finally I haven't touched distil because you've been less then forthcoming
about it's own code. I realize it's just using zip tech to bundle it and thus
I can unpack it and read it but:

  A) That is a gigantic pain in the ass to do just because for reasons unknown
     you've been unwilling to publish it's source.
  B) I actually *did* that once, and found that there is no OSS license
     attached to the code and instead it's "All Rights Reserved" at which point
     I trashed the files.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140201/19f70df4/attachment.sig>