[Python-ideas] Sharing modules across virtualenvs

[Phyks]

Thanks for the explanations! Will look at distuil-sig for further infos.

-- 
Phyks

Le 12/02/2016 11:53, Nick Coghlan a écrit :
> On 12 February 2016 at 20:00, Phyks <phyks at phyks.me> wrote:
>> Hi,
>>
>> I was wondering recently about the way virtualenvs work, and did not
>> find much information on the design choices. Sorry if this is not the
>> right mailing-list to post about it, but it seemed to be.
>>
>> As far as I understand, virtualenvs work by modifying the PYTHONHOME and
>> then the folders in which to look up to import modules.
>>
>> Are there any specific reasons for this implementation, and not a
>> "/lib"-like implementation, putting the various versions of the module
>> in a single folder, with a version naming, and symlinking the up-to-date
>> one?
> 
> Symlinking unfortunately doesn't solve the problem for a couple of reasons:
> 
> * symlinks are still basically unusable on Windows
> * symlinks and https://packaging.python.org/en/latest/specifications/#recording-installed-distributions
> don't get along
> 
> Module sharing between virtual environments *could* still be
> implemented based on *.pth files, but:
> 
> * you'd need to update an installer (probably pip) to handle shared
> installs and uninstalls
> * you'd have a refcounting problem on deciding when it was OK to
> uninstall versions
> * you'd run into the startup performance problems that can arise with
> .pth files and adding lots of entries to sys.path
> 
> And even if you do manage to solve all those problems, at the end of
> it, you'll have essentially reinvented some of the core components of
> nix: https://nixos.org/nix/
> 
>> Then, virtualenvs could simply be applied as filters on this specific
>> folder, to hide part of the available module. Contrary to the current
>> implementation, this means building (and storing, although storage is
>> said to be cheap) large Python modules only once for your system, and
>> share them.
> 
> The implicit wheel cache used in recent versions of pip tackles that
> problem in a different way (the built wheel fields are cached, so you
> don't have to build them again, but they still get unpacked separately
> into each virtualenv that needs them)
> 
> Cheers,
> Nick.
> 
> P.S. This is more a distutils-sig topic than a python-ideas one, so if
> you'd like to learn more or explore the idea further, I'd suggest
> following up there
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>