Upstream Patch to inherit virtual environment ?
Hello Python Ideas, Would there be any interest in upstreaming a patch allowing virtualenv to be inherited in core python ? I'm working with a company that uses such an internal patch and would be ok with me spending some time trying to upstream it. When creating a venv, this let you point to "parents" venvs that will be used to search for existing packages; thus potentially making the creation of large venvs with a common base faster, and decreasing disk space usage. I am aware that there are solution around like, https://virtualenvwrapper.readthedocs.io/en/latest/command_ref.html#add2virt..., or modifying PYTHONPATH , but my question is more about whether such a change has a chance of being accepted in Core Python; and whether there is a need for a pep,, or simply a request for enhancement on bpo and a pull-request. Thanks, -- Matthias
First thoughts: - that's added complexity. - is that reproducible? - how can I detect that that's the case? - a standard way might be easiest - what controls prevent one user from modifying the parent venv and thereby everything that depends upon it? - would there need to be modifications to tools, or do you just manually set VIRTUAL_ENV to the parent? - are there strange effects e.g. when v1 has pkg.this but v2 has removed pkg.this and then `impory pkg.this` still succeeds? On Tue, Oct 20, 2020, 2:24 PM Matthias Bussonnier < bussonniermatthias@gmail.com> wrote:
On Wed, Oct 21, 2020 at 5:24 AM Matthias Bussonnier <bussonniermatthias@gmail.com> wrote:
The main thing a proposal like this needs is a thread on python-ideas, so you're on the right track already :) If I'm understanding you correctly, this means I could do something like: $ python3 -m venv env --see-also ../master-env and then any packages installed in master-env would be visible inside this environment too? Seems pretty useful to me. What's the mechanism by which a venv announces its parent, and does this work correctly if the binary is invoked directly? Currently, if you invoke /.../.../.../env/bin/python3, it will use the lib directory from that venv; with this plan, it would need to add the parent also. ChrisA
On Tue, 20 Oct 2020 at 20:16, Chris Angelico <rosuav@gmail.com> wrote:
I'd imagine that the "parent" could be put in `pyvenv.cfg`, and the interpreter startup code would (somehow) add that to `sys.path`. Which is mostly straightforward, but `sys.path` setup is (I believe) fairly deeply entangled with interpreter startup. I'd assume that the effect of the mechanism would be just like any other means of adding entries to sys.path - that's both the intended layering benefits, and the potentially bad/confusing ones like different versions of a package shadowing each other. It's also worth noting that tools like pip would probably struggle with managing linked environments like this. I suspect that's actually a far bigger problem than the core mechanism. Pip is already struggling with the amount of complexity it has to deal with, I don't think we'd be able to add something like this. So you might have to manually manage the "layers", or build a tool that wraps pip and handles the layering. Paul
On Wed, Oct 21, 2020 at 6:30 AM Paul Moore <p.f.moore@gmail.com> wrote:
That's what I'd have figured too, but interpreter startup does make it more complicated.
There's already the --system-site-packages flag to venv; I'm not sure how pip behaves with that - does "pip freeze" list system-installed packages too? Whatever it is, I would just assume that the parent venv is between the current one and the system path, with similar behaviour. What happens if the parent has a parent? What happens if you create circular parentage (by changing the config file on the parent to have the child as a parent - a classic time travel paradox)? Assuming these edge cases can be resolved in some well-defined way (even if that's "throw an error if you try that"), I'm +1 on this proposal. ChrisA
Chris Said:
Yes, it does allows multiple parent as well.
There is indeed as Paul thought later a file that describe the parent(s) that can be edited after the fact, I belive this is useful when one want to test a dev version of a software across many base environment so that you only need the testing software + the dev version on the child env, and switch easily. I did not went into much details about the curent implementation both as I'm still getting familiar with the patch, and that I expect that anyway exact behavior woudl lead to bikeshed, and I expect would be better suited for a later discussion. As for the executable I would have to check, but I belive it woudl do the right thing, or at least we can make it do so I believe. Paul Said:
Yes, the patch introduce an env variable and flags to deactivate all this search in the layering. But layering is opt-in so I woudl expect it to be used only in advanced usage with folks who know what they are doing. Chris Said:
Currently the patch does a depth first search, so parents can have parents.
Again don't want to get into details, but I'm unsure this is a problem, the env you activate will have higher precedence over the other EnvA(parent=EnvB): - Foo==1.0 - Bar==1.0 and EnvB(parent=EnvA): - Foo==2.0 - Qux==2.0 Activate A -> Foo==1.0, Bar==1.0, Qux==2.0 Activate B -> Foo==2.0, Bar==1.0, Qux==2.0 The only library that is present in both (Foo), and you get the version of the child you activate.
Thanks, I'll clean up the patch that have unrelated changes, and port it to current master. I opened an issue on BPO (https://bugs.python.org/issue42101). I expect the exact details to change during upstreaming this; and if need be we can do a several stages where we first stay conservative decide on edge cases later. Matthias
First thoughts: - that's added complexity. - is that reproducible? - how can I detect that that's the case? - a standard way might be easiest - what controls prevent one user from modifying the parent venv and thereby everything that depends upon it? - would there need to be modifications to tools, or do you just manually set VIRTUAL_ENV to the parent? - are there strange effects e.g. when v1 has pkg.this but v2 has removed pkg.this and then `impory pkg.this` still succeeds? On Tue, Oct 20, 2020, 2:24 PM Matthias Bussonnier < bussonniermatthias@gmail.com> wrote:
On Wed, Oct 21, 2020 at 5:24 AM Matthias Bussonnier <bussonniermatthias@gmail.com> wrote:
The main thing a proposal like this needs is a thread on python-ideas, so you're on the right track already :) If I'm understanding you correctly, this means I could do something like: $ python3 -m venv env --see-also ../master-env and then any packages installed in master-env would be visible inside this environment too? Seems pretty useful to me. What's the mechanism by which a venv announces its parent, and does this work correctly if the binary is invoked directly? Currently, if you invoke /.../.../.../env/bin/python3, it will use the lib directory from that venv; with this plan, it would need to add the parent also. ChrisA
On Tue, 20 Oct 2020 at 20:16, Chris Angelico <rosuav@gmail.com> wrote:
I'd imagine that the "parent" could be put in `pyvenv.cfg`, and the interpreter startup code would (somehow) add that to `sys.path`. Which is mostly straightforward, but `sys.path` setup is (I believe) fairly deeply entangled with interpreter startup. I'd assume that the effect of the mechanism would be just like any other means of adding entries to sys.path - that's both the intended layering benefits, and the potentially bad/confusing ones like different versions of a package shadowing each other. It's also worth noting that tools like pip would probably struggle with managing linked environments like this. I suspect that's actually a far bigger problem than the core mechanism. Pip is already struggling with the amount of complexity it has to deal with, I don't think we'd be able to add something like this. So you might have to manually manage the "layers", or build a tool that wraps pip and handles the layering. Paul
On Wed, Oct 21, 2020 at 6:30 AM Paul Moore <p.f.moore@gmail.com> wrote:
That's what I'd have figured too, but interpreter startup does make it more complicated.
There's already the --system-site-packages flag to venv; I'm not sure how pip behaves with that - does "pip freeze" list system-installed packages too? Whatever it is, I would just assume that the parent venv is between the current one and the system path, with similar behaviour. What happens if the parent has a parent? What happens if you create circular parentage (by changing the config file on the parent to have the child as a parent - a classic time travel paradox)? Assuming these edge cases can be resolved in some well-defined way (even if that's "throw an error if you try that"), I'm +1 on this proposal. ChrisA
Chris Said:
Yes, it does allows multiple parent as well.
There is indeed as Paul thought later a file that describe the parent(s) that can be edited after the fact, I belive this is useful when one want to test a dev version of a software across many base environment so that you only need the testing software + the dev version on the child env, and switch easily. I did not went into much details about the curent implementation both as I'm still getting familiar with the patch, and that I expect that anyway exact behavior woudl lead to bikeshed, and I expect would be better suited for a later discussion. As for the executable I would have to check, but I belive it woudl do the right thing, or at least we can make it do so I believe. Paul Said:
Yes, the patch introduce an env variable and flags to deactivate all this search in the layering. But layering is opt-in so I woudl expect it to be used only in advanced usage with folks who know what they are doing. Chris Said:
Currently the patch does a depth first search, so parents can have parents.
Again don't want to get into details, but I'm unsure this is a problem, the env you activate will have higher precedence over the other EnvA(parent=EnvB): - Foo==1.0 - Bar==1.0 and EnvB(parent=EnvA): - Foo==2.0 - Qux==2.0 Activate A -> Foo==1.0, Bar==1.0, Qux==2.0 Activate B -> Foo==2.0, Bar==1.0, Qux==2.0 The only library that is present in both (Foo), and you get the version of the child you activate.
Thanks, I'll clean up the patch that have unrelated changes, and port it to current master. I opened an issue on BPO (https://bugs.python.org/issue42101). I expect the exact details to change during upstreaming this; and if need be we can do a several stages where we first stay conservative decide on edge cases later. Matthias
participants (4)
-
Chris Angelico -
Matthias Bussonnier -
Paul Moore -
Wes Turner