repeatable and updatable virtualenv
Hi All, How can I set up a virtualenv in such a way that I can version control the packages and their versions that are installed in it? The answer would appear to a custom bootstrap script, but how would I think update an existing virtualenv when I added or removed packages or changed the version of a specific package? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
How can I set up a virtualenv in such a way that I can version control the packages and their versions that are installed in it?
You'd use pip together with virtualenv (which is easy because every virtualenv nowadays comes with pip pre-installed), and you'd version-control a requirements file containing package names and versions.
The answer would appear to a custom bootstrap script, but how would I think update an existing virtualenv when I added or removed packages or changed the version of a specific package?
If all the package names in your requirements.txt are version-pinned, then "$VENV/bin/pip install -r requirements.txt" will ensure that you have those versions of all those packages installed. The only thing it won't do for you is remove packages that are no longer in requirements.txt; you'd need to do that yourself with "pip uninstall packagename" - not that having extra packages installed matters, generally. Alternatively, if you want to be paranoid and you don't mind things being a little slower (though not much if you have a local PIP_DOWNLOAD_CACHE), you just create a new virtualenv each time you deploy. Virtualenvs are disposable, only the requirements.txt matters. Carl
Carl Meyer wrote:
Chris Withers wrote:
How can I set up a virtualenv in such a way that I can version control the packages and their versions that are installed in it?
You'd use pip together with virtualenv (which is easy because every virtualenv nowadays comes with pip pre-installed), and you'd version-control a requirements file containing package names and versions.
Okay, but I think you said at pycon that pip won't work with binary or .egg distributions? Does it play nicely with multiple indexes, some of which may be private and require http authentication?
If all the package names in your requirements.txt are version-pinned, then "$VENV/bin/pip install -r requirements.txt" will ensure that you have those versions of all those packages installed. The only thing it won't do for you is remove packages that are no longer in requirements.txt; you'd need to do that yourself with "pip uninstall packagename" - not that having extra packages installed matters, generally.
Well, it can matter if you accidentally end up relying on them...
Alternatively, if you want to be paranoid and you don't mind things being a little slower (though not much if you have a local PIP_DOWNLOAD_CACHE), you just create a new virtualenv each time you deploy.
That won't necessarily fly... where would you put data files? outside the virtualenv? feels a bit weird to me, but that may just be a hysterical raisin... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Okay, but I think you said at pycon that pip won't work with binary or .egg distributions?
True; IIUC the plan is to eventually support some form of binary package for Windows, but currently binary packages are not supported at all. I don't have the need for binary packages; if I did I imagine my first step would involve version-controlling a simple shell script using easy_install or something. Or if that became unwieldy I'd look to a buildout-based solution?
Does it play nicely with multiple indexes, some of which may be private and require http authentication?
Multiple indexes, sure; you can replace PyPI or layer an additional index on top of it, or just use --find-links with a directory. HTTP auth I don't believe is supported, though I'm not completely sure; wouldn't be hard to add, pip just uses urllib2.
Alternatively, if you want to be paranoid and you don't mind things being a little slower (though not much if you have a local PIP_DOWNLOAD_CACHE), you just create a new virtualenv each time you deploy.
That won't necessarily fly... where would you put data files? outside the virtualenv? feels a bit weird to me, but that may just be a hysterical raisin...
Nothing goes inside my virtualenvs except code installed via pip. The virtualenv is a deployment artifact, not a folder container for my project. My project code checkout, any data files, anything I care about goes in its own location outside the virtualenv. I'm sure that does seem odd coming from a buildout background, but AFAICS there's no good reason to ever manually touch files inside the virtualenv directory, and lots of good reasons not to. Carl
Carl Meyer wrote:
Chris Withers wrote:
Okay, but I think you said at pycon that pip won't work with binary or .egg distributions?
True; IIUC the plan is to eventually support some form of binary package for Windows, but currently binary packages are not supported at all.
This is, quite frankly, ridiculous, and not just because of Windows. So, say I produce a binary egg for mysqldb, pip's position appears to be that I'm wrong, and I should have to make sure all the build dependencies are present on my whole cluster of identical machines. Same goes for lxml, and anything else with complex build dependencies. Really, wtf? ;-)
don't have the need for binary packages; if I did I imagine my first step would involve version-controlling a simple shell script using easy_install or something. Or if that became unwieldy I'd look to a buildout-based solution?
Right, so I guess I'm back to buildout...
Does it play nicely with multiple indexes, some of which may be private and require http authentication?
Multiple indexes, sure; you can replace PyPI or layer an additional index on top of it, or just use --find-links with a directory. HTTP auth I don't believe is supported, though I'm not completely sure; wouldn't be hard to add, pip just uses urllib2.
Hmm, but it's not there yes, and it is with buildout...
Nothing goes inside my virtualenvs except code installed via pip. The virtualenv is a deployment artifact, not a folder container for my project. My project code checkout, any data files, anything I care about goes in its own location outside the virtualenv. I'm sure that does seem odd coming from a buildout background, but AFAICS there's no good reason to ever manually touch files inside the virtualenv directory, and lots of good reasons not to.
Yeah, I think this is just a style thing :-) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
At 09:35 AM 3/17/2010 +0000, Chris Withers wrote:
Carl Meyer wrote:
Chris Withers wrote:
Okay, but I think you said at pycon that pip won't work with binary or .egg distributions? True; IIUC the plan is to eventually support some form of binary package for Windows, but currently binary packages are not supported at all.
This is, quite frankly, ridiculous, and not just because of Windows. So, say I produce a binary egg for mysqldb, pip's position appears to be that I'm wrong, and I should have to make sure all the build dependencies are present on my whole cluster of identical machines. Same goes for lxml, and anything else with complex build dependencies. Really, wtf? ;-)
Ian might not be here to say it, but if he were, I imagine he might say, "patches welcome". ;-) Seriously, though, I've posted samples here more than once of the basic code needed to extract an .egg into a pip-style configuration, and easy_install has code that will turn windows .exe files into .egg files. Those parts aren't hard at all, relatively speaking. What somebody would still need to write, are the pip-specific bits to wrap all that in an installation process (including finding the downloadables in the first place).
P.J. Eby wrote:
At 09:35 AM 3/17/2010 +0000, Chris Withers wrote:
Carl Meyer wrote:
Chris Withers wrote:
Okay, but I think you said at pycon that pip won't work with binary or .egg distributions? True; IIUC the plan is to eventually support some form of binary package for Windows, but currently binary packages are not supported at all.
This is, quite frankly, ridiculous, and not just because of Windows. So, say I produce a binary egg for mysqldb, pip's position appears to be that I'm wrong, and I should have to make sure all the build dependencies are present on my whole cluster of identical machines. Same goes for lxml, and anything else with complex build dependencies. Really, wtf? ;-)
Ian might not be here to say it, but if he were, I imagine he might say, "patches welcome". ;-)
It's likely easier for me to get buildout to do what I need, which is a shame, since I *do* like the feel of virtualenv...
Seriously, though, I've posted samples here more than once of the basic code needed to extract an .egg into a pip-style configuration,
I'm not following you, I don't want to extract any eggs, I just want to have dependencies satisfied by distributions that may well be binary eggs.
What somebody would still need to write, are the pip-specific bits to wrap all that in an installation process (including finding the downloadables in the first place).
I thought pip just used setuptools/easyinstall for that anyway? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Tue, 2010-03-16 at 08:26 +0000, Chris Withers wrote:
Hi All,
How can I set up a virtualenv in such a way that I can version control the packages and their versions that are installed in it?
The answer would appear to a custom bootstrap script, but how would I think update an existing virtualenv when I added or removed packages or changed the version of a specific package?
we use a virtualenv bootstrap script that set an virtualenv+buildout project, using the same directory as base for both ones, we dont find any problems with this setup in our projects. then we only set the versions for the packages that we like in the buildout recipe. another advantage is that any command installed in the bin/ dir for the buildout can be called directly if we activate the virtualenv. our script if find an bootstrap.py and buildout.cfg files in the same dir that we run the virtualenv bootstrap script, build the system, ideal for deployments in production -- Yonsy Solis Aureal Systems (mov) 989-124-141 (www) http://www.aureal.pe
Chris> Hi All, How can I set up a virtualenv in such a way that I can Chris> version control the packages and their versions that are Chris> installed in it? requirements.txt and pip has already been mentioned. If you need to version control particular packages, you might use pip install -e and install them directly from e.g. Git/SVN/...
On Tue, Mar 16, 2010 at 2:26 AM, Chris Withers <chris@simplistix.co.uk> wrote:
Hi All,
How can I set up a virtualenv in such a way that I can version control the packages and their versions that are installed in it?
The answer would appear to a custom bootstrap script, but how would I think update an existing virtualenv when I added or removed packages or changed the version of a specific package?
You could try envbuilder; I find it very helpful for setting up development environments repeatably when developing several related packages ('er, 'module distributions') simultaneously. It basically just creates a virtualenv, checks out/clones whatever you specify in the .env confiig file, and runs setup.py develop on each of the checked out packages into the virtualenv. Also, it provides a way to apply commands across all the checked out packages (like 'update' or 'status') http://pypi.python.org/pypi/envbuilder/0.1.4 I think a similar functionality can be had using buildout with mr.developer, but I have not tried it yet. However as far as I know, buildout does not create a virtualenv (yet).
On Fri, Mar 19, 2010 at 8:31 AM, Brad Allen <bradallen137@gmail.com> wrote:
http://pypi.python.org/pypi/envbuilder/0.1.4
I think a similar functionality can be had using buildout with mr.developer, but I have not tried it yet. However as far as I know, buildout does not create a virtualenv (yet).
For the record, the most recent stable version is 0.2.2: http://pypi.python.org/pypi/envbuilder/0.2.2 There's also a development version in github: Also, another option could be to use a pip requirements file.
2010/3/19 Brad Allen <bradallen137@gmail.com>
On Tue, Mar 16, 2010 at 2:26 AM, Chris Withers <chris@simplistix.co.uk> wrote:
Hi All,
How can I set up a virtualenv in such a way that I can version control the packages and their versions that are installed in it?
The answer would appear to a custom bootstrap script, but how would I think update an existing virtualenv when I added or removed packages or changed the version of a specific package?
You could try envbuilder; I find it very helpful for setting up development environments repeatably when developing several related packages ('er, 'module distributions') simultaneously.
It basically just creates a virtualenv, checks out/clones whatever you specify in the .env confiig file, and runs setup.py develop on each of the checked out packages into the virtualenv. Also, it provides a way to apply commands across all the checked out packages (like 'update' or 'status')
http://pypi.python.org/pypi/envbuilder/0.1.4
I think a similar functionality can be had using buildout with mr.developer, but I have not tried it yet. However as far as I know, buildout does not create a virtualenv (yet).
I'm currently trying more or less the same thing by combining buildout with virtualenv and pip. So far it's working without problem except the installation of develop eggs (see my other message). Buildout can create virtualenv without problem (see http://pypi.python.org/pypi/rjm.recipe.venv/0.7) and can use pip to install editable package (see http://pypi.python.org/pypi/gp.recipe.pip) under version control (CVS, SVN, Mercurial...) Now if only I can solve the develop eggs issue, I would be very happy !
Saint Germain wrote:
I'm currently trying more or less the same thing by combining buildout with virtualenv and pip.
This doesn't seem like a sane thing to do. These tools attempt to address the same problems in different ways. Attempting to mash them together as it appears you have does nothing but result in an overcomplicated solution that's likely to be more fragile than if you'd worked with either buildout or pip/virtualenv on their own. cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
2010/3/22 Chris Withers <chris@simplistix.co.uk>:
I'm currently trying more or less the same thing by combining buildout with virtualenv and pip.
This doesn't seem like a sane thing to do.
These tools attempt to address the same problems in different ways. Attempting to mash them together as it appears you have does nothing but result in an overcomplicated solution that's likely to be more fragile than if you'd worked with either buildout or pip/virtualenv on their own.
Well I mainly wanted to set up environment using virtualenv+pip and it worked pretty well. However there were a lot of steps involved and I wanted to automated the process. So I just get the virtualenv + pip recipe and voilà ! Nothing overcomplicated, and a fallback to manually install everything is always possible (indeed with the recipe, it's very simple, and they are just a mean to automatically call virtualenv or pip). I thought that buildout was designed for just that, am I wrong ?
Saint Germain wrote:
2010/3/22 Chris Withers <chris@simplistix.co.uk>:
I'm currently trying more or less the same thing by combining buildout with virtualenv and pip. This doesn't seem like a sane thing to do.
These tools attempt to address the same problems in different ways. Attempting to mash them together as it appears you have does nothing but result in an overcomplicated solution that's likely to be more fragile than if you'd worked with either buildout or pip/virtualenv on their own.
Well I mainly wanted to set up environment using virtualenv+pip and it worked pretty well.
Then why not just use virtualenv+pip with a custom bootstrap script?!
I thought that buildout was designed for just that, am I wrong ?
It's a bit like using Subversion to manage a set of CVS checkouts... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
2010/3/22 Chris Withers <chris@simplistix.co.uk>:
Saint Germain wrote:
2010/3/22 Chris Withers <chris@simplistix.co.uk>:
I'm currently trying more or less the same thing by combining buildout with virtualenv and pip.
This doesn't seem like a sane thing to do.
These tools attempt to address the same problems in different ways. Attempting to mash them together as it appears you have does nothing but result in an overcomplicated solution that's likely to be more fragile than if you'd worked with either buildout or pip/virtualenv on their own.
Well I mainly wanted to set up environment using virtualenv+pip and it worked pretty well.
Then why not just use virtualenv+pip with a custom bootstrap script?!
Well because buildout makes all this very efficiently and simply. All the options (find-links, newest, etc...) are already there. Basically if I try to make a custom bootsrap script, I will end up with more or less the same buildout.cfg, I will have to manage all the options and the interaction, etc. For the error, I will have to have my own error management. Even for bootstrapping, I will have to download the same ez_setup in order to easy_install what I want. In addition, if I want to introduce some external C libraries, putting this in my bootstrap will be some kind of hack. In buildout, I just have to call the cmmi recipe, and voilà. Really I don't see the point of trying to create from scratch my own bootstrap script when buildout make all this so simple.
I thought that buildout was designed for just that, am I wrong ?
It's a bit like using Subversion to manage a set of CVS checkouts...
I don't think that the comparison is fair.
On Mon, Mar 22, 2010 at 09:59, Chris Withers <chris@simplistix.co.uk> wrote:
Saint Germain wrote:
I'm currently trying more or less the same thing by combining buildout with virtualenv and pip.
This doesn't seem like a sane thing to do.
Sure it does. It's just not very useful in most cases. Buildout by itself works fine to isolate the python installation from the stuff installed. But if you want to also isolate the buildout from what's installed in the python, then you can use virtualenv for that. Unusual usecase, but sane. :)
Lennart Regebro wrote:
On Mon, Mar 22, 2010 at 09:59, Chris Withers <chris@simplistix.co.uk> wrote:
Saint Germain wrote:
I'm currently trying more or less the same thing by combining buildout with virtualenv and pip. This doesn't seem like a sane thing to do.
Sure it does. It's just not very useful in most cases. Buildout by itself works fine to isolate the python installation from the stuff installed.
Well, no, using a virtualenv to wrap a buildout is far from insane, especially until Gary's work to make site-packages optionally excluded from the search path lands! However, the OP's recommended recipes appear to just use buildout to drive virtualenv, ignoring all of buildout's python package management. That seems insane... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Lennart Regebro wrote:
On Mon, Mar 22, 2010 at 09:59, Chris Withers <chris@simplistix.co.uk> wrote:
Saint Germain wrote:
I'm currently trying more or less the same thing by combining buildout with virtualenv and pip. This doesn't seem like a sane thing to do.
Sure it does. It's just not very useful in most cases. Buildout by itself works fine to isolate the python installation from the stuff installed.
Well, no, using a virtualenv to wrap a buildout is far from insane, especially until Gary's work to make site-packages optionally excluded from the search path lands!
However, the OP's recommended recipes appear to just use buildout to drive virtualenv, ignoring all of buildout's python package management. That seems insane...
I wrote the rjm.recipe.venv recipe. All it does is turn the buildout root directory into a virtualenv, so you can use virtualenv to wrap the buildout w/o having to take an extra step, or worry about whether or not one of your downstream users has forgotten to do so. That being said, I also like the idea of gp.recipe.pip. I'll admit that it _is_ a bit insane. But the pip/virtualenv combination fits my brain better than buildout. I don't much care for zc.recipe.egg, I prefer virtualenv's approach of mimicking Python's package management mechanisms, so that I don't have to context switch between my sandboxed environments and the way Python works at the system level. gp.recipe.pip is nowhere near as battle-hardened as zc.recipe.egg, nor does it support every use case that zc.recipe.egg supports, but I can certainly understand the motivation. -r
participants (10)
-
Brad Allen -
Carl Meyer -
Chris Withers -
Jason Baker -
Lennart Regebro -
P.J. Eby -
Rob Miller -
Saint Germain -
Suno Ano -
Yonsy Solis