Sharing modules across virtualenvs
Hi, I was wondering recently about the way virtualenvs work, and did not find much information on the design choices. Sorry if this is not the right mailing-list to post about it, but it seemed to be. As far as I understand, virtualenvs work by modifying the PYTHONHOME and then the folders in which to look up to import modules. Are there any specific reasons for this implementation, and not a "/lib"-like implementation, putting the various versions of the module in a single folder, with a version naming, and symlinking the up-to-date one? Then, virtualenvs could simply be applied as filters on this specific folder, to hide part of the available module. Contrary to the current implementation, this means building (and storing, although storage is said to be cheap) large Python modules only once for your system, and share them. I do not doubt there were great reasons to choose to move the lookup path, however. But I do not know which one they are. Thanks! -- Phyks
On 12 February 2016 at 20:00, Phyks <phyks@phyks.me> wrote:
Symlinking unfortunately doesn't solve the problem for a couple of reasons: * symlinks are still basically unusable on Windows * symlinks and https://packaging.python.org/en/latest/specifications/#recording-installed-d... don't get along Module sharing between virtual environments *could* still be implemented based on *.pth files, but: * you'd need to update an installer (probably pip) to handle shared installs and uninstalls * you'd have a refcounting problem on deciding when it was OK to uninstall versions * you'd run into the startup performance problems that can arise with .pth files and adding lots of entries to sys.path And even if you do manage to solve all those problems, at the end of it, you'll have essentially reinvented some of the core components of nix: https://nixos.org/nix/
The implicit wheel cache used in recent versions of pip tackles that problem in a different way (the built wheel fields are cached, so you don't have to build them again, but they still get unpacked separately into each virtualenv that needs them) Cheers, Nick. P.S. This is more a distutils-sig topic than a python-ideas one, so if you'd like to learn more or explore the idea further, I'd suggest following up there -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 12 February 2016 at 20:00, Phyks <phyks@phyks.me> wrote:
Symlinking unfortunately doesn't solve the problem for a couple of reasons: * symlinks are still basically unusable on Windows * symlinks and https://packaging.python.org/en/latest/specifications/#recording-installed-d... don't get along Module sharing between virtual environments *could* still be implemented based on *.pth files, but: * you'd need to update an installer (probably pip) to handle shared installs and uninstalls * you'd have a refcounting problem on deciding when it was OK to uninstall versions * you'd run into the startup performance problems that can arise with .pth files and adding lots of entries to sys.path And even if you do manage to solve all those problems, at the end of it, you'll have essentially reinvented some of the core components of nix: https://nixos.org/nix/
The implicit wheel cache used in recent versions of pip tackles that problem in a different way (the built wheel fields are cached, so you don't have to build them again, but they still get unpacked separately into each virtualenv that needs them) Cheers, Nick. P.S. This is more a distutils-sig topic than a python-ideas one, so if you'd like to learn more or explore the idea further, I'd suggest following up there -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (2)
-
Nick Coghlan -
Phyks