[Distutils] Installing packages using pip

[Erik Bray]

On Fri, Nov 13, 2015 at 3:09 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Nov 13, 2015 12:00 PM, "Alexander Walters" <tritium-list at sdamon.com>
> wrote:
>>
>> import pip
>> pip.install(PACKAGESPEC)
>>
>> something like that?
>
> This would be extremely handy if it could be made to work reliably... But
> I'm skeptical about whether it can be made to work reliably. Consider all
> the fun things that could happen once you start upgrading packages while
> python is running, and might e.g. have half of an upgraded package already
> loaded into memory. It's like the reloading problem but even more so.

Sorry to resurrect an old thread, but I have an idea about how to do
this somewhat safely, at least insofar as the running interpreter is
concerned.  It's still a terrible idea.  Not such a terrible idea in
principle, but as a practical matter in the context of Python it's
probably a bad idea because it uses yet-another-.pth-hack.

Consider a "partial install", wherein pip installs all files into a
non-imported subdirectory of the target site-packages, along with a
.pth file.  This distribution is then considered "partially installed"
in that the files are their (whether extracted from a wheel, or
installed via distutils and the appropriate --root option or similar).
For example, consider running

>>> pip.install('requests')

It would be up to the pip.install() command to determine whether or
not the requests distribution was already installed.  If it's not
install it would proceed as normal.  For now I'm assuming the user
would still have to manually run `import requests` after this.
Auto-import would be nice, but is a separate issue.

Now, if requests were already installed and imported we don't want to
clobber the existing requests running in the interpreter.  pip would
install install into the relevant site-packages:

<...>/site-packages/requests-2.8.1.part/
    requests/
    requests-2.8.1.dist-info/
<...>/site-packages/requests-2.8.1.part.pth

The .part/ directory contains the results of the partial installation
(for example the contents of the wheel, for wheel installs).  The
.part.pth file is trickier, but could be something like this:

$ cat requests-2.8.1.part.pth
import inspect, shutil, sys, os, atexit;p =
inspect.currentframe().f_locals['sitedir'];part = os.path.join(p,
'requests-2.8.1.part');files = os.path.isdir(part) and
os.listdir(part);files and list(map(lambda s, d, f:
(sys.modules['shutil'].rmtree(os.path.join(d, f),
sys.modules['shutil'].move(os.path.join(s, f), os.path.join(d, f))),
[part] * len(files), [p] * len(files), files));os.rmdir(part);pth =
part + '.pth';os.path.isfile(pth) and atexit.register(os.unlink,
os.path.abspath(pth))

This rifles through the contents of requests.2.8.1.part, deletes any
existing directories in the parent site-packages of the same name,
completes the install by moving the contents of the .part/ directory
into the correct location and then deletes the .part/ directory.  The
.part.pth later deletes itself.

By the time the user restarts the interpreter and runs `import
requests` this will be completed.  Obvious it would have to be
communicated to the user that to upgrade an existing package they will
have to restart the interpreter which is less than ideal, but relates
to a deeper limitation of Python that they should get used to anyways.
At least this would enable in-process installs/upgrades.

There are of course all kinds of problems with this solution too.  It
should perhaps only work in a virtualenv and/or .local site-packages
(or at least somewhere that the user will have write permissions on
the next interpreter run), and probably other error handling too.  The
above .pth file could also be simplified by invoking a function in pip
to complete any partial installs.

Erik