[Distutils] restart-in-venv (was: PEX at Twitter (re: PEX - Twitter's multi-platform executable archive format for Python))

[Vinay Sajip]

On Tue, 4/2/14, Paul Moore <p.f.moore at gmail.com> wrote:

> The big problem with this type of thing is that pip tends to be used
> on lots of systems with sometimes *extremely* obscure and odd
> setups - what some of the Linux distros do to a standard Python
> install amazes me (I'm sure they have good reasons, of course...).
> So it's not so much a case of "scary dark corners" as lots of
> unanticipated and extremely difficult to reproduce use cases that
> we have to support but where the only knowledge of what
> workarounds are needed is encapsulated in the code.

You've just restated "scary dark corners" in a different way which
sounds like there's more to it technically. But there' s still no hard
data on the table!

Surely, with all the convolutions that these distros go through, can one
still assume that pip-the-program-that-they-all-run assumes control via
some notional main program, which is passed some arguments? If
we can't assume this, please tell me why not. If we can, then it would
be like this:

def main():
    # Do stuff with sys.argv

Now, distil has a similar entry point (as innumerable other scripts do)
and all I did to restart in venv was to look for a -e argument and process
it before the main event:

def main():
    if '-e' found in sys.argv:
        validate the passed parameter, prepare a new command line
        using the interpreter from the venv and the sys.argv passed in,
        but minus the -e bit, since we've dealt with that here
        call subprocess with that command line (the restart) 
        return - our work here is done
    # Do stuff with sys.argv

It doesn't get much simpler. Of course there could be bugs in the
code called in the if suite, but it's not doing a complicated job so it
should be easy to identify and eradicate them. Now, are you saying
that the bit of code represented by that if suite, which only has to
process sys.argv, look for an identifiable venv in the file system and
check that it seems to have a runnable Python interpreter, can be
bamboozled by something distros do to their Python installations?
I'm having trouble buying that as an argument, unless there is more
hard data on the table. What "unanticipated and extremely difficult
to reproduce use cases" could interfere with the operations in that
if suite? It's hard to argue against voodoo, but anyone on this list
could put me right with a particular circumstance that I haven't
thought of, but that they have run into, which undermines my
argument. It's not uncommon to see the "well ... we're not quite
sure what's happening in there ... better play it safe", but that
seems out of place in an engineering context.

Regards,

Vinay Sajip