, the behaviour i aim for would be moslty like
virtualenv but without the file duplication.

I beleive nix could also benefit from parts of such a mechanism.

-- Ronny


Am Montag, den 30.10.2017, 20:35 +0100 schrieb Freddy Rietdijk:

Hi Ronny,

What you describe here is, as you know, basically what the Nix

package manager does. You could create something similar specifically

for Python, like e.g. `ied` is for Node [2], or Spack, which is

written in Python. But then how are you going to deal with other

system libraries, and impurities? And you will have to deal with

them, because depending on how you configure Python packages that

depend on them (say a `numpy`), their output will be different. Or

would you choose to ignore this?

Freddy

[1] https://nixos.org/nix/

[2] https://github.com/alexanderGugel/ied

[3] https://spack.io/

On Mon, Oct 30, 2017 at 8:16 PM, RonnyPfannschmidt <opensource@ronnyp

fannschmidt.de> wrote:

Hi everyone,

since a while now various details of installing python packages in

virtualenvs caused me grief

a) typically each tox folder in a project is massive, and has a lot

of

duplicate files, recreating them, managing and iterating them takes

quite a while

b) for nicely separated deployments, each virtualenv for an

application

takes a few hundred megabytes - that quickly can saturate disk

space

even if a reasonable amount was reserved

c) installation and recreation of virtualenvs with the same set of

packages takes quite a while (even with pip caches this is slow,

and

there is no good reason to avoid making it completely

instantaneous)

in order to elevate those issues i would like to propose a new

installation layout,

where instead of storing each distribution in every python all

distributions would share a storage, and each individual

environment

would only have references to the packages that where

"installed/activated" for them

this would massively reduce time required to create the contents of

the

environments and also the space required

since blindly expanding sys.path would lead to similar performance

issues as where seen with setuptools/buildout multi-version

installs,

this mechanism would also need a element on sys.meta_path that

handles

inexpensive dispatch to the toplevels and metadata files of each

packages (off hand i would assume linear walking of hundreds of

entries

simply isn't that effective)

however there would be need for some experimentation to see what

tradeoff is sensible there

I hope this mail will spark enough discussion to enable the

creation of

a PEP and a prototype.

Best, Ronny

_______________________________________________

Distutils-SIG maillist  -  Distutils-SIG@python.org

https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig