[Distutils] [proposal] shared distribution installations

[RonnyPfannschmidt]

Hi everyone,

since a while now various details of installing python packages in
virtualenvs caused me grief

a) typically each tox folder in a project is massive, and has a lot of
duplicate files, recreating them, managing and iterating them takes
quite a while
b) for nicely separated deployments, each virtualenv for an application
takes a few hundred megabytes - that quickly can saturate disk space
even if a reasonable amount was reserved
c) installation and recreation of virtualenvs with the same set of
packages takes quite a while (even with pip caches this is slow, and
there is no good reason to avoid making it completely instantaneous)

in order to elevate those issues i would like to propose a new
installation layout,
where instead of storing each distribution in every python all
distributions would share a storage, and each individual environment
would only have references to the packages that where
"installed/activated" for them

this would massively reduce time required to create the contents of the
environments and also the space required

since blindly expanding sys.path would lead to similar performance
issues as where seen with setuptools/buildout multi-version installs,
this mechanism would also need a element on sys.meta_path that handles
inexpensive dispatch to the toplevels and metadata files of each
packages (off hand i would assume linear walking of hundreds of entries
simply isn't that effective)

however there would be need for some experimentation to see what
tradeoff is sensible there

I hope this mail will spark enough discussion to enable the creation of
a PEP and a prototype.


Best, Ronny