[Distutils] [proposal] shared distribution installations

Ronny Pfannschmidt opensource at ronnypfannschmidt.de
Mon Oct 30 17:09:51 EDT 2017


Hi Freddy,

im well aware what nix currently does for python packages and suffered
my fair share from it.

What i want to do is simply store those wheels that pip would first
generate and unpack into each environment into a location where each
environment shares the unpacked files more directly

im not going to expand uppon my perceived shortcomings of nix as i know
,it since its irrelevant to this discussion and not something i have
the time and motivation to fix.

as far as impurities go, the behaviour i aim for would be moslty like
virtualenv but without the file duplication.

I beleive nix could also benefit from parts of such a mechanism.

-- Ronny


Am Montag, den 30.10.2017, 20:35 +0100 schrieb Freddy Rietdijk:
> Hi Ronny,
> 
> What you describe here is, as you know, basically what the Nix
> package manager does. You could create something similar specifically
> for Python, like e.g. `ied` is for Node [2], or Spack, which is
> written in Python. But then how are you going to deal with other
> system libraries, and impurities? And you will have to deal with
> them, because depending on how you configure Python packages that
> depend on them (say a `numpy`), their output will be different. Or
> would you choose to ignore this? 
> 
> Freddy
> 
> [1] https://nixos.org/nix/
> [2] https://github.com/alexanderGugel/ied
> [3] https://spack.io/
> 
> On Mon, Oct 30, 2017 at 8:16 PM, RonnyPfannschmidt <opensource at ronnyp
> fannschmidt.de> wrote:
> > Hi everyone,
> > 
> > since a while now various details of installing python packages in
> > virtualenvs caused me grief
> > 
> > a) typically each tox folder in a project is massive, and has a lot
> > of
> > duplicate files, recreating them, managing and iterating them takes
> > quite a while
> > b) for nicely separated deployments, each virtualenv for an
> > application
> > takes a few hundred megabytes - that quickly can saturate disk
> > space
> > even if a reasonable amount was reserved
> > c) installation and recreation of virtualenvs with the same set of
> > packages takes quite a while (even with pip caches this is slow,
> > and
> > there is no good reason to avoid making it completely
> > instantaneous)
> > 
> > in order to elevate those issues i would like to propose a new
> > installation layout,
> > where instead of storing each distribution in every python all
> > distributions would share a storage, and each individual
> > environment
> > would only have references to the packages that where
> > "installed/activated" for them
> > 
> > this would massively reduce time required to create the contents of
> > the
> > environments and also the space required
> > 
> > since blindly expanding sys.path would lead to similar performance
> > issues as where seen with setuptools/buildout multi-version
> > installs,
> > this mechanism would also need a element on sys.meta_path that
> > handles
> > inexpensive dispatch to the toplevels and metadata files of each
> > packages (off hand i would assume linear walking of hundreds of
> > entries
> > simply isn't that effective)
> > 
> > however there would be need for some experimentation to see what
> > tradeoff is sensible there
> > 
> > I hope this mail will spark enough discussion to enable the
> > creation of
> > a PEP and a prototype.
> > 
> > 
> > Best, Ronny
> > 
> > 
> > 
> > _______________________________________________
> > Distutils-SIG maillist  -  Distutils-SIG at python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
> 
> 


More information about the Distutils-SIG mailing list