Hi,
On 31 October 2017 at 05:22, Nick Coghlan
On 31 October 2017 at 05:16, RonnyPfannschmidt < opensource@ronnypfannschmidt.de> wrote:
Hi everyone,
since a while now various details of installing python packages in virtualenvs caused me grief
a) typically each tox folder in a project is massive, and has a lot of duplicate files, recreating them, managing and iterating them takes quite a while b) for nicely separated deployments, each virtualenv for an application takes a few hundred megabytes - that quickly can saturate disk space even if a reasonable amount was reserved c) installation and recreation of virtualenvs with the same set of packages takes quite a while (even with pip caches this is slow, and there is no good reason to avoid making it completely instantaneous)
Those are issues that buildout has solved long before pip was even around, but they rely on sys.path expansion that Ronny found objectionable due to performance issues. I don't think the performance issues are that problematic (and wasn't there some work on Python 3 that made import faster even with long sys.paths?).
[...]
However, there's another approach that specifically tackles the content duplication problem, which would require a new installation layout as you suggest, but could still rely on *.pth files to make it implicitly compatible with existing packages and applications and existing Python runtime versions.
That approach is to create an install tree somewhere that looks like this:
_shared-packages/ <normalised-package-name>/ <release-version>/ <version-details>.dist-info/ <installed-files>
Instead of installing full packages directly into a venv the way pip does, an installer that worked this way would instead manage a <normalised-package-name>.pth file that indicated "_shared-packages/<normalised-package-name>/<release-version>" should be added to sys.path.
This solution is nice, but preserves the long sys.path that Ronny wanted to avoid in the first place. Another detail that needs mentioning is that, for .pth based sys.path manipulation to work, the <installed-files> would need to be all the files from purelib and platlib directories from wheels mashed together instead of a simple unpacking of the wheel (though I guess the .pth file could add both purelib and platlib subfolders to sys.path...) Another possibility that avoids the issue of long.syspath is to use this layout but with symlink farms instead of either sys.path manipulation or conda-like hard-linking. Symlinks would preserve better filesystem size visibility that Ronny wanted while allowing the layout above to contain wheels that were simply unzipped. In Windows, where symlinks require admin privileges (though this is changing https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/), an option could be provided for using hard links instead (which never require elevated privileges). Using symlinks into the above layout preserves all advantages and drawbacks Nick mentioned other than the sys.path expansion. Regards, Leo