[Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

[Nick Coghlan]

On 1 February 2014 22:20, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:
>
> On Sat, 1/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> My point is that doing it the way virtualenv/pip did avoided a bunch
>> of design work and associated testing, and reduced the
>> opportunities for bugs - when you're trying to get things done with
>> limited resources, that's a sensible engineering trade-off to make.
>
> A "bunch of design work" makes it seem a lot more complicated
> than it really is. Your suggestion comes across like an ex post facto
> rationalisation of a decision which was perhaps more truly based
> on social concerns than technical ones.

I have no idea how most of the internal design decisions on pip were
made (and I neither need nor particularly want to know - that would be
rather missing the point of collaborative development). However,
you're trying to make out that the distil approach is so clearly
superior that you don't understand why anyone would ever have
implemented it differently, and I'm trying to point out that there's
an entirely plausible answer to that question that doesn't involve
assuming poor engineering trade-offs on the part of the pip
development team.

> Note that I've developed distil as part of my volunteer activities - it
> doesn't pay any of my bills - and on my own. And you're telling me
> about how to get the best out of "limited resources"? :-)

No, things we do in our free time are the *best* opportunities to
investigate things like that, because they're not necessarily goal
oriented - they're more about satisfying our curiousity. Goal oriented
development is a *terrible* environment for exploration, because we
have to ruthlessly prune activities that don't obviously move us
closer to our goal.

Consider the following two engineering design options:

A. We install an independent copy of the installer in every virtual
environment, guaranteeing full isolation from the system Python and
permitting the use of different installer versions for different
projects, all using the exact same isolation mechanism as is used for
any Python component

B. We design a separate mechanism that allows a system installation of
the installer to easily be used to install software into virtual
environments, but requiring that the system and all virtual
environments be using the same version of the installer, and needing
to come up with a distinct targeting mechanism separate from the
activation mechanism already provided by virtual environments

Now *a priori*, you don't know how complex B is going to be. On the
other hand, you already *know* A will work, because you already have
working virtual environments.

So, from a scope of isolation point of view, A has the advantage: the
installer itself is isolated in addition to the installed components.

And from a design uncertainty point of view, A also has the advantage:
there is *no* design uncertainty, because you're just using the
already-known-to-work virtual environment mechanism.

So in a goal-oriented "provide isolation of Python applications"
context, there's no *reason* to even explore B: it has more design
uncertainty, and offers a lower level of isolation (because the
installer is still shared).

Now, you *have* had time to explore option B in distil, and have
discovered that it actually wasn't that hard after all. Cool, that's
great and would be a nice feature to have in pip as well. However, the
fact that *having already done it*, you discovered it wasn't all that
difficult, doesn't change the fact that the level of effort involved
was still greater than the nonexistent amount of work needed to get
the existing solution working once virtual environments are available
(and I noticed that you cut the part of the email pointing out the
concrete technical benefits of being able to isolate the installer in
addition to the installed components).

It's one thing to disagree with a design decision after we have made
the absolutely best possible case that we can for why that decision
was the right choice, and still failing to convince ourselves. It's
something else entirely to dismiss someone else's use case as invalid
and *then* argue that an arguably simpler design (that doesn't handle
the dismissed use case and was still only hypothetical at the time the
decision was made) would have been a better choice.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia