pip installing scripts into another virtualenv
As part of a package management tool https://github.com/alexbecker/dotlock, I'm trying to use pip to install python packages into a virtualenv, from python code (via subprocess), into a different virtualenv than the virtualenv my python process is running in (if any). I have this *mostly* working with:
pip install --prefix [virtualenv-path] --ignore-installed --no-build-isolation
However, installed scripts break because the scrips automatically get prepended with a shebang pointing to the python interpreter pip was run under.
Is there a way around this behavior? Am I crazy to even try to install into a different virtualenv? Or do I have to re-architect my code to call pip in the target virtualenv (which may require me forcing pip to be installed, depending on what versions of python I choose to support)?
Sincerely,
Alex Becker
Yes, why not simply invoke the pip installed in the virtualenv into which you want to install the packages? It doesn't seem like you'd need to "re-architect" anything insomuch as simply change "pip" in your argument list to the path to the pip inside the virtualenv you're targeting.
--Chris
On Fri, Sep 14, 2018 at 12:15 AM, Alex Becker alcubecker@gmail.com wrote:
As part of a package management tool, I'm trying to use pip to install python packages into a virtualenv, from python code (via subprocess), into a different virtualenv than the virtualenv my python process is running in (if any). I have this *mostly* working with:
pip install --prefix [virtualenv-path] --ignore-installed --no-build-isolation
However, installed scripts break because the scrips automatically get prepended with a shebang pointing to the python interpreter pip was run under.
Is there a way around this behavior? Am I crazy to even try to install into a different virtualenv? Or do I have to re-architect my code to call pip in the target virtualenv (which may require me forcing pip to be installed, depending on what versions of python I choose to support)?
Sincerely,
Alex Becker
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/C...
The main downside to invoking pip in the virtualenv is that it means I have to have pip installed within the virtualenv, which is something I haven't wanted to commit to in the interest of creating slim virtualenvs for bundling with apps. But you are probably right and I need to accept that.
On Fri, Sep 14, 2018 at 12:43 AM Chris Jerdonek chris.jerdonek@gmail.com wrote:
Yes, why not simply invoke the pip installed in the virtualenv into which you want to install the packages? It doesn't seem like you'd need to "re-architect" anything insomuch as simply change "pip" in your argument list to the path to the pip inside the virtualenv you're targeting.
--Chris
On Fri, Sep 14, 2018 at 12:15 AM, Alex Becker alcubecker@gmail.com wrote:
As part of a package management tool, I'm trying to use pip to install python packages into a virtualenv, from python code (via subprocess),
into a
different virtualenv than the virtualenv my python process is running in
(if
any). I have this *mostly* working with:
pip install --prefix [virtualenv-path] --ignore-installed --no-build-isolation
However, installed scripts break because the scrips automatically get prepended with a shebang pointing to the python interpreter pip was run under.
Is there a way around this behavior? Am I crazy to even try to install
into
a different virtualenv? Or do I have to re-architect my code to call pip
in
the target virtualenv (which may require me forcing pip to be installed, depending on what versions of python I choose to support)?
Sincerely,
Alex Becker
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/C...
Isn't virtualenv's default behavior to install pip automatically when a virtualenv is created? https://virtualenv.pypa.io/en/stable/userguide/#usage (That is, I don't think you need to "force" pip to be installed, but rather simply rely on default behavior.)
--Chris
On Fri, Sep 14, 2018 at 12:50 AM, Alex Becker alcubecker@gmail.com wrote:
The main downside to invoking pip in the virtualenv is that it means I have to have pip installed within the virtualenv, which is something I haven't wanted to commit to in the interest of creating slim virtualenvs for bundling with apps. But you are probably right and I need to accept that.
On Fri, Sep 14, 2018 at 12:43 AM Chris Jerdonek chris.jerdonek@gmail.com wrote:
Yes, why not simply invoke the pip installed in the virtualenv into which you want to install the packages? It doesn't seem like you'd need to "re-architect" anything insomuch as simply change "pip" in your argument list to the path to the pip inside the virtualenv you're targeting.
--Chris
On Fri, Sep 14, 2018 at 12:15 AM, Alex Becker alcubecker@gmail.com wrote:
As part of a package management tool, I'm trying to use pip to install python packages into a virtualenv, from python code (via subprocess), into a different virtualenv than the virtualenv my python process is running in (if any). I have this *mostly* working with:
pip install --prefix [virtualenv-path] --ignore-installed --no-build-isolation
However, installed scripts break because the scrips automatically get prepended with a shebang pointing to the python interpreter pip was run under.
Is there a way around this behavior? Am I crazy to even try to install into a different virtualenv? Or do I have to re-architect my code to call pip in the target virtualenv (which may require me forcing pip to be installed, depending on what versions of python I choose to support)?
Sincerely,
Alex Becker
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/C...
I am confused. If you’re installing things with subprocess, and is using virtual environments anyway, wouldn’t it be simpler to use the pip inside the virtual environment directly?
Instead of using a (random) pip to install into the environment, you can instead do
[virtualenv-path]/[script-prefix]/python -m pip install …
The script prefix would be bin in most cases, or Scripts on Windows. You can dig into the source of venv or virtualenv to find the exact logic and use it, but in practice it is basically enough to test both locations until you find the correct Python executable. (This is also Pipenv’s approach, essentially.)
I wouldn’t say it’s crazy to pip-install into a foreign virtual environment, but can’t recommend it either. pip does a lot of runtime detection when choosing an artifact to install a package. So unless you’re installing directly from URL, there is no guarantee you are installing the correct artifact for a given package, or even choosing the correct version (when you lock the requirements). Pipenv actually used a similar approach in its initial design, but found it to be very error-prone. Reading the project description, however, it seems like you’re locking “specific distributions” for a given package, so I guess this this is not necessarily a problem? I am not sure how you can choose the correct distribution/artifact without using the in-environment pip though (I didn’t read the implementation).
Finally, if all you really want is to download/unpack/install the locked artifact to use, maybe you can even do away with virtual environments altogether. There was a discussion around ditching virtual environments altogether in favour of a node/npm-like setup at this year’s PyCon (I didn’t attend)[1] that you might find interesting.
[1]: https://lwn.net/Articles/757354/ https://lwn.net/Articles/757354/
TP
On 14/9, 2018, at 15:15, Alex Becker alcubecker@gmail.com wrote:
As part of a package management tool https://github.com/alexbecker/dotlock, I'm trying to use pip to install python packages into a virtualenv, from python code (via subprocess), into a different virtualenv than the virtualenv my python process is running in (if any). I have this *mostly* working with:
pip install --prefix [virtualenv-path] --ignore-installed --no-build-isolation
However, installed scripts break because the scrips automatically get prepended with a shebang pointing to the python interpreter pip was run under.
Is there a way around this behavior? Am I crazy to even try to install into a different virtualenv? Or do I have to re-architect my code to call pip in the target virtualenv (which may require me forcing pip to be installed, depending on what versions of python I choose to support)?
Sincerely,
Alex Becker
Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/C...
On Fri, 14 Sep 2018 at 08:16, Alex Becker alcubecker@gmail.com wrote:
Is there a way around this behavior? Am I crazy to even try to install into a different virtualenv? Or do I have to re-architect my code to call pip in the target virtualenv (which may require me forcing pip to be installed, depending on what versions of python I choose to support)?
The simple answer here is that this isn't something that pip supports, nor is it something that anyone has ever considered adding support for. So you're basically on your own if you want to avoid using the pip in the target virtualenv.
Paul
To add to these responses, Tzu Ping and I have tried this in a myriad of creative ways and mostly haven’t been successful while relying on pip’s cli. The right answer is to invoke the virtualenv pip, it’s just easier
-D
Dan Ryan // pipenv maintainer gh: @techalchemy
On Sep 14, 2018, at 4:09 AM, Paul Moore p.f.moore@gmail.com wrote:
On Fri, 14 Sep 2018 at 08:16, Alex Becker alcubecker@gmail.com wrote: Is there a way around this behavior? Am I crazy to even try to install into a different virtualenv? Or do I have to re-architect my code to call pip in the target virtualenv (which may require me forcing pip to be installed, depending on what versions of python I choose to support)?
The simple answer here is that this isn't something that pip supports, nor is it something that anyone has ever considered adding support for. So you're basically on your own if you want to avoid using the pip in the target virtualenv.
Paul
Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/Y...
On Fri, 14 Sep 2018 at 18:40, Dan Ryan dan@danryan.co wrote:
To add to these responses, Tzu Ping and I have tried this in a myriad of creative ways and mostly haven’t been successful while relying on pip’s cli. The right answer is to invoke the virtualenv pip, it’s just easier
pip can also be uninstalled afterwards if the aim is to minimise the size of the shipped virtualenv, and any future updates will be handled by replacing the entire virtualenv rather than modifying it in place.
Cheers, Nick.
participants (6)
-
Alex Becker -
Chris Jerdonek -
Dan Ryan -
Nick Coghlan -
Paul Moore -
Tzu-ping Chung