[Distutils] PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python)

[Vinay Sajip]

On Sat, 1/2/14, Nick Coghlan <ncoghlan at gmail.com> wrote:

> FWIW, installing into a venv from outside it works fine

That's what I'm saying :-)

> However, it's substantially *harder* to explain to people how
> to use it correctly that way.

But distil installs into venvs (both PEP 405 and virtualenv)
without needing to be in there. It's as simple as

distil -e path/to/venv install XYZ

So I'm not sure what you mean by "use correctly" or what
exactly is hard to explain. Distil doesn't do anything special -
after all, installing a distribution just means moving
certain files into certain locations. If you know that you're
installing into one of (a) a specific venv, or (b) user
site-packages, or (c) system site-packages, then you know
what the installation locations are without doing anything
especially clever, don't you?

Of course, building and installation are commingled in
distutils and setuptools - that's "build-from-source-is-
all-you need" myopia right there - but I don't see how that
prevents installation-from-without or forces the usage of
installation-from-within, at least at a fundamental level.

> In theory you could change activation so that it also
> affected the default install locations, but the advantage
> of just having them installed per venv is that you're relying
> more on the builtin Python path machinery rather
> than adding something new.

But virtualenv solves an analogous problem when creating
venvs with different interpreters by reinvoking itself with a
specific interpreter. Distil does something similar, but that's
more to do with the fact that code in setup.py sometimes
takes different paths depending on which version of Python
is running it.

> So while it's wasteful of disk space and

Disk space is cheap, so that's not the issue, it's the
lack of elegance.

> Avoiding work by leveraging existing capabilities is a time
> honoured engineering tradition, even when the simple way
> isn't the most elegant way.

I'm pretty pragmatic, so I agree with this point in general,
but in this case I think you're side-stepping the issue of
technical debt. We're talking about file-system manipulations,
after all, not anything more esoteric than that. Any approach
which perpetuates the "python setup.py install" way of doing
things is, to me, not a good thing, which I think has been
recognised from the good old days when Tarek started
taking a fresh look at packaging.

> I have now realised the much-reviled (for good reason) *.pth
> files actually have a legitimate use case in allowing API
> compatible versions of packages to be shared between
> multiple virtual environments

Do you mean ref files, as discussed on import-sig?

Sure, but that's a tangential issue in my view. Worthwhile
pursuing, certainly, but an incremental improvement over what
we have now (and actually less necessary in the parallel
universe where wheels are more like jars and not reviled for
that just because of "Java - ewww").

Regards,

Vinay Sajip