[Python-ideas] Sharing modules across virtualenvs

[Nick Coghlan]

On 12 February 2016 at 20:00, Phyks <phyks at phyks.me> wrote:
> Hi,
>
> I was wondering recently about the way virtualenvs work, and did not
> find much information on the design choices. Sorry if this is not the
> right mailing-list to post about it, but it seemed to be.
>
> As far as I understand, virtualenvs work by modifying the PYTHONHOME and
> then the folders in which to look up to import modules.
>
> Are there any specific reasons for this implementation, and not a
> "/lib"-like implementation, putting the various versions of the module
> in a single folder, with a version naming, and symlinking the up-to-date
> one?

Symlinking unfortunately doesn't solve the problem for a couple of reasons:

* symlinks are still basically unusable on Windows
* symlinks and https://packaging.python.org/en/latest/specifications/#recording-installed-distributions
don't get along

Module sharing between virtual environments *could* still be
implemented based on *.pth files, but:

* you'd need to update an installer (probably pip) to handle shared
installs and uninstalls
* you'd have a refcounting problem on deciding when it was OK to
uninstall versions
* you'd run into the startup performance problems that can arise with
.pth files and adding lots of entries to sys.path

And even if you do manage to solve all those problems, at the end of
it, you'll have essentially reinvented some of the core components of
nix: https://nixos.org/nix/

> Then, virtualenvs could simply be applied as filters on this specific
> folder, to hide part of the available module. Contrary to the current
> implementation, this means building (and storing, although storage is
> said to be cheap) large Python modules only once for your system, and
> share them.

The implicit wheel cache used in recent versions of pip tackles that
problem in a different way (the built wheel fields are cached, so you
don't have to build them again, but they still get unpacked separately
into each virtualenv that needs them)

Cheers,
Nick.

P.S. This is more a distutils-sig topic than a python-ideas one, so if
you'd like to learn more or explore the idea further, I'd suggest
following up there

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia