[Distutils] [proposal] shared distribution installations

[Leonardo Rochael Almeida]

Hi,

On 31 October 2017 at 05:22, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 31 October 2017 at 05:16, RonnyPfannschmidt <
> opensource at ronnypfannschmidt.de> wrote:
>
>> Hi everyone,
>>
>> since a while now various details of installing python packages in
>> virtualenvs caused me grief
>>
>> a) typically each tox folder in a project is massive, and has a lot of
>> duplicate files, recreating them, managing and iterating them takes
>> quite a while
>> b) for nicely separated deployments, each virtualenv for an application
>> takes a few hundred megabytes - that quickly can saturate disk space
>> even if a reasonable amount was reserved
>> c) installation and recreation of virtualenvs with the same set of
>> packages takes quite a while (even with pip caches this is slow, and
>> there is no good reason to avoid making it completely instantaneous)
>>
>
Those are issues that buildout has solved long before pip was even around,
but they rely on sys.path expansion that Ronny found objectionable due to
performance issues.

I don't think the performance issues are that problematic (and wasn't there
some work on Python 3 that made import faster even with long sys.paths?).


> [...]
>
> However, there's another approach that specifically tackles the content
> duplication problem, which would require a new installation layout as you
> suggest, but could still rely on *.pth files to make it implicitly
> compatible with existing packages and applications and existing Python
> runtime versions.
>
> That approach is to create an install tree somewhere that looks like this:
>
>     _shared-packages/
>         <normalised-package-name>/
>             <release-version>/
>                 <version-details>.dist-info/
>                 <installed-files>
>
> Instead of installing full packages directly into a venv the way pip does,
> an installer that worked this way would instead manage a
> <normalised-package-name>.pth file that indicated
> "_shared-packages/<normalised-package-name>/<release-version>" should be
> added to sys.path.
>

This solution is nice, but preserves the long sys.path that Ronny wanted to
avoid in the first place.

Another detail that needs mentioning is that, for .pth based sys.path
manipulation to work, the <installed-files> would need to be all the files
from purelib and platlib directories from wheels mashed together instead of
a simple unpacking of the wheel (though I guess the .pth file could add
both purelib and platlib subfolders to sys.path...)

Another possibility that avoids the issue of long.syspath is to use this
layout but with symlink farms instead of either sys.path manipulation or
conda-like hard-linking.

Symlinks would preserve better filesystem size visibility that Ronny wanted
while allowing the layout above to contain wheels that were simply unzipped.

In Windows, where symlinks require admin privileges (though this is changing
<https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/>),
an option could be provided for using hard links instead (which never
require elevated privileges).

Using symlinks into the above layout preserves all advantages and drawbacks
Nick mentioned other than the sys.path expansion.

Regards,

Leo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20171031/8a951982/attachment-0001.html>