[Distutils] restart-in-venv (was: PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python))

[Carl Meyer]

Hi Vinay,

On 02/01/2014 09:25 AM, Vinay Sajip wrote:
> On Sat, 1/2/14, Donald Stufft <donald at stufft.io> wrote:
>> So perhaps it isn't that pip is lacking a useful feature, it's that
>> pip tried it, as you did with distil, and due to pip's much larger
>> user base it got a much more thorough real world experience
>> with it and it turned out to be ultimately more trouble than it was
>> worth? It could also have been a problem with how pip wrote it
>> of course, and perhaps you've discovered a better way of
>> writing it. However at the time it's obvious that the pain it was
>> causing was enough to cause the pip developers to break
>> backwards compatibility to remove that feature.
> 
>> Why exactly was it removed? Well I can't find the reasoning
>> beyond that commit so we'd need to get Carl or perhaps Jannis
>> to come and tell us if they still remember. However it's clearly not
>> that pip didn't just try or think of this, it did and it ultimately removed it.
> 
> Sure, and if they did find some problem which is also present in
> distil (i.e. due to a fundamental drawback of the approach) then I'll put
> my thinking cap back on. It'll be worth having that problem on record
> to avoid any repetition of work in the future.

Sadly I can't offer that specific problem, because AFAIK we never
tracked it down. It was a recurring problem in IRC where people would
report using -E and pip would fail to take its action within the
indicated virtualenv, instead impacting their global environment. After
trying and failing a number of times to isolate and reproduce that
problem, I decided not to continue spending time on that, and that it
would be more robust to rely on the natural isolation provided by just
using pip inside the env, which required no additional code to debug and
maintain. (Pip had already been installed inside every virtualenv for
quite a while by then.)

I regret the way I handled that particular backwards-incompatibility
(not providing a deprecation path), but I think the technical choice was
a good one. Identifying code that is a bug-report-magnet and choosing a
viable approach where that code can be removed entirely is a sound
technical choice, not a matter of "politics" or "religion" (I had no
political or religious motivations in removing pip -E).

As far as I can see, you haven't presented any technical issues with the
current approach, just a feeling that it's inelegant (is this perhaps a
religious feeling? :P). My feeling differs; I find it elegant to rely
only on the virtualenv's built-in isolation, and inelegant to require
extra restart-in-env code.

Carl