- are there strange effects e.g. when v1 has pkg.this but v2 has removed pkg.this and then `impory pkg.this` still succeeds?

parent: pip install pkg==1.0 child: pip install pkg==2.0 child: python -c "import pkg.this"

On Tue, Oct 20, 2020, 2:24 PM Matthias Bussonnier < bussonniermatthias@gmail.com> wrote:

...

Hello Python Ideas,

Would there be any interest in upstreaming a patch allowing virtualenv to be inherited in core python ? I'm working with a company that uses such an internal patch and would be ok with me spending some time trying to upstream it.

When creating a venv, this let you point to "parents" venvs that will be used to search for existing packages; thus potentially making the creation of large venvs with a common base faster, and decreasing disk space usage.

I am aware that there are solution around like, https://virtualenvwrapper.readthedocs.io/en/latest/command_ref.html#add2virt..., or modifying PYTHONPATH , but my question is more about whether such a change has a chance of being accepted in Core Python; and whether there is a need for a pep,, or simply a request for enhancement on bpo and a pull-request.

Thanks, -- Matthias _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/KTIYUE... Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, 20 Oct 2020 at 20:16, Chris Angelico <rosuav@gmail.com> wrote:

What's the mechanism by which a venv announces its parent, and does this work correctly if the binary is invoked directly? Currently, if you invoke /.../.../.../env/bin/python3, it will use the lib directory from that venv; with this plan, it would need to add the parent also.

I'd imagine that the "parent" could be put in `pyvenv.cfg`, and the interpreter startup code would (somehow) add that to `sys.path`. Which is mostly straightforward, but `sys.path` setup is (I believe) fairly deeply entangled with interpreter startup. I'd assume that the effect of the mechanism would be just like any other means of adding entries to sys.path - that's both the intended layering benefits, and the potentially bad/confusing ones like different versions of a package shadowing each other. It's also worth noting that tools like pip would probably struggle with managing linked environments like this. I suspect that's actually a far bigger problem than the core mechanism. Pip is already struggling with the amount of complexity it has to deal with, I don't think we'd be able to add something like this. So you might have to manually manage the "layers", or build a tool that wraps pip and handles the layering. Paul

Chris Said:

If I'm understanding you correctly, this means I could do something like:

$ python3 -m venv env --see-also ../master-env

and then any packages installed in master-env would be visible inside this environment too? Seems pretty useful to me.

Yes, it does allows multiple parent as well.

What's the mechanism by which a venv announces its parent, and does this work correctly if the binary is invoked directly? Currently, if you invoke /.../.../.../env/bin/python3, it will use the lib directory from that venv; with this plan, it would need to add the parent also.

There is indeed as Paul thought later a file that describe the parent(s) that can be edited after the fact, I belive this is useful when one want to test a dev version of a software across many base environment so that you only need the testing software + the dev version on the child env, and switch easily. I did not went into much details about the curent implementation both as I'm still getting familiar with the patch, and that I expect that anyway exact behavior woudl lead to bikeshed, and I expect would be better suited for a later discussion. As for the executable I would have to check, but I belive it woudl do the right thing, or at least we can make it do so I believe. Paul Said:

I'd imagine that the "parent" could be put in pyvenv.cfg, and the interpreter startup code would (somehow) add that to sys.path. Which is mostly straightforward, but sys.path setup is (I believe) fairly deeply entangled with interpreter startup. I'd assume that the effect of the mechanism would be just like any other means of adding entries to sys.path - that's both the intended layering benefits, and the potentially bad/confusing ones like different versions of a package shadowing each other.

It's also worth noting that tools like pip would probably struggle with managing linked environments like this. I suspect that's actually a far bigger problem than the core mechanism. Pip is already struggling with the amount of complexity it has to deal with, I don't think we'd be able to add something like this. So you might have to manually manage the "layers", or build a tool that wraps pip and handles the layering.

Yes, the patch introduce an env variable and flags to deactivate all this search in the layering. But layering is opt-in so I woudl expect it to be used only in advanced usage with folks who know what they are doing. Chris Said:

There's already the --system-site-packages flag to venv; I'm not sure how pip behaves with that - does "pip freeze" list system-installed packages too? Whatever it is, I would just assume that the parent venv is between the current one and the system path, with similar behaviour.

What happens if the parent has a parent?

Currently the patch does a depth first search, so parents can have parents.

What happens if you create circular parentage (by changing the config file on the parent to have the child as a parent - a classic time travel paradox)?

Again don't want to get into details, but I'm unsure this is a problem, the env you activate will have higher precedence over the other EnvA(parent=EnvB): - Foo==1.0 - Bar==1.0 and EnvB(parent=EnvA): - Foo==2.0 - Qux==2.0 Activate A -> Foo==1.0, Bar==1.0, Qux==2.0 Activate B -> Foo==2.0, Bar==1.0, Qux==2.0 The only library that is present in both (Foo), and you get the version of the child you activate.

Assuming these edge cases can be resolved in some well-defined way (even if that's "throw an error if you try that"), I'm +1 on this proposal.

Thanks, I'll clean up the patch that have unrelated changes, and port it to current master. I opened an issue on BPO (https://bugs.python.org/issue42101). I expect the exact details to change during upstreaming this; and if need be we can do a several stages where we first stay conservative decide on edge cases later. Matthias