We have a suite of tools in the 'lazr' namespace that we've begun to release as open source. All of these packages are maintained under bzr and it's really nice to be able to add this to our setup.py:
setup_requires=['setuptools_bzr']
This extends setuptools so that we can use bzr instead of the built-in revision control systems. With that, 'python setup.py sdist' pulls in the right stuff. All of our packages are buildout based.
The problem is that when we buildout the package, the download cache is never consulted and setuptools_bzr is /always/ downloaded. This is a non-starter for our production environments, which are not allowed to hit the external 'net during build time. Because of this, we've had to disable the setup_requires. We can kind of make things work with MANIFEST.ins but I'd prefer to be able to use setup_requires. Otherwise, um, what's the point of having this plugin?
I'm not sure whether this is a bug in setuptools, distutils, or buildout, but since y'all are here, we should be able to figure it out. Or do you not think this is a bug?
If you agree it's a bug, and we can agree where it's a bug, I'd be willing to put some time into fixing this, if we can get a timely update released in the affected package.
Thanks, -Barry
At 06:09 PM 6/22/2009 -0400, Barry Warsaw wrote:
We have a suite of tools in the 'lazr' namespace that we've begun to release as open source. All of these packages are maintained under bzr and it's really nice to be able to add this to our setup.py:
setup_requires=['setuptools_bzr']
This extends setuptools so that we can use bzr instead of the built-in revision control systems. With that, 'python setup.py sdist' pulls in the right stuff. All of our packages are buildout based.
The problem is that when we buildout the package, the download cache is never consulted and setuptools_bzr is /always/ downloaded.
I don't know what the "download cache" is, so I can't speak to that. However, what I can say is that if you're using "sdist" for distribution, you probably don't need that setup_requires line in the first place.
When you build an sdist using setuptools, it includes a pre-generated file manifest, along with a setup.cfg that forces the right revision tag information in the project's version number (if applicable). This is specifically designed so that the person building your sdist doesn't need the same revision control system installed.
In short, the only person who needs setuptools_bzr installed is the person who's generating the sdists. So, feel free to drop it from setup_requires and be happy. ;-)
Sorry for the delay in responding and the lack of clarity in the original message...
On Jun 22, 2009, at 11:40 PM, P.J. Eby wrote:
I don't know what the "download cache" is, so I can't speak to that. However, what I can say is that if you're using "sdist" for distribution, you probably don't need that setup_requires line in the first place.
download cache is a buildout thing.
When you build an sdist using setuptools, it includes a pre- generated file manifest, along with a setup.cfg that forces the right revision tag information in the project's version number (if applicable). This is specifically designed so that the person building your sdist doesn't need the same revision control system installed.
In short, the only person who needs setuptools_bzr installed is the person who's generating the sdists. So, feel free to drop it from setup_requires and be happy. ;-)
Here's what the setuptools docs say:
(Note: projects listed in setup_requires will NOT be automatically installed on the system where the setup script is being run. They are simply downloaded to the setup directory if they're not locally available already. If you want them to be installed, as well as being available when the setup script is run, you should add them to install_requires and setup_requires.)
So I get that setuptools is doing what it's supposed to do. I also get that it's (usually) not too hard to write a MANIFEST.in that pretty much does what you need it to do. I'm bothered by the fact that this puts us in a bind where we want to use setup_requires but really can't. After further thought though, the "problem" is probably in buildout, because it shouldn't be downloading setup_requires packages if they are in its download cache.
-Barry
At 05:02 PM 6/26/2009 -0400, Barry Warsaw wrote:
When you build an sdist using setuptools, it includes a pre- generated file manifest, along with a setup.cfg that forces the right revision tag information in the project's version number (if applicable). This is specifically designed so that the person building your sdist doesn't need the same revision control system installed.
In short, the only person who needs setuptools_bzr installed is the person who's generating the sdists. So, feel free to drop it from setup_requires and be happy. ;-)
Here's what the setuptools docs say:
(Note: projects listed in setup_requires will NOT be automatically installed on the system where the setup script is being run. They are simply downloaded to the setup directory if they're not locally available already. If you want them to be installed, as well as being available when the setup script is run, you should add them to install_requires and setup_requires.)
So I get that setuptools is doing what it's supposed to do. I also get that it's (usually) not too hard to write a MANIFEST.in that pretty much does what you need it to do. I'm bothered by the fact that this puts us in a bind where we want to use setup_requires but really can't. After further thought though, the "problem" is probably in buildout, because it shouldn't be downloading setup_requires packages if they are in its download cache.
You seem to be ignoring the part where *you don't need setup_requires* in order to get a correct manifest on the target system, as long as they're using an sdist.
In other words, the only people who need setuptools_bzr are the people *making* sdists, not the people *using* sdists. You can therefore omit it from your setup_requires, if I understand your current situation correctly.
On Jun 26, 2009, at 7:50 PM, P.J. Eby wrote:
You seem to be ignoring the part where *you don't need setup_requires* in order to get a correct manifest on the target system, as long as they're using an sdist.
In other words, the only people who need setuptools_bzr are the people *making* sdists, not the people *using* sdists. You can therefore omit it from your setup_requires, if I understand your current situation correctly.
I get that. The problem is that there's nothing like an 'sdist_requires' which would list packages needed to make the sdist. I'm not the only one making dists for upload, and I'd like to be able to specify this requirement in the setup.py so that other people making dists would have less dependency on their environment (i.e. repeatablility). So maybe setup_requires isn't the right thing to use, but it's the only hook to hang this hat on (AFAICT).
But maybe there is and I've just overlooked it.
-Barry
Barry:
What is the download cache?
I have a guess. If you are installing with the setuptools "install" command, then the things which are "install_requires" get installed into some target location, but things which are build_requires get installed into the PWD. (If you don't specify a target, then it is the system site-packages. In my build system [1], we typically don't use "install" and instead use "develop" with a --prefix argument which specifies the target. I assume virtualenv does something similar.)
Therefore, a later build will have its install_requires already satisfied (if it is using the same installation target), but its build_requirements will not be satisfied.
If that's what's going on then a solution might involve pre- installing setuptools_bzr onto the build system, or patching setuptools to specify a location for build requirements and re-using that location.
Regards,
Zooko
Hi Zooko,
On Jun 23, 2009, at 11:42 AM, Zooko Wilcox-O'Hearn wrote:
What is the download cache?
It's where buildout caches packages it downloads and it's very handy because we can tell buildout to use a pre-defined download cache and to never download anything from the intarwebs. Except that this doesn't work for setup_requires packages, so that seems like a bug in buildout. We really don't want to add those to install_requires because the clients of our libraries don't care.
I have a guess. If you are installing with the setuptools "install" command, then the things which are "install_requires" get installed into some target location, but things which are build_requires get installed into the PWD.
Right, that's another thing I don't particularly like about setup_requires. I don't like those packages polluting my pwd.
(If you don't specify a target, then it is the system site- packages. In my build system [1], we typically don't use "install" and instead use "develop" with a --prefix argument which specifies the target. I assume virtualenv does something similar.)
I think so. We use buildout, which has the moral equivalent of 'develop'.
Therefore, a later build will have its install_requires already satisfied (if it is using the same installation target), but its build_requirements will not be satisfied.
Right.
If that's what's going on then a solution might involve pre- installing setuptools_bzr onto the build system, or patching setuptools to specify a location for build requirements and re-using that location.
We could pre-install setuptools_bzr if we built an Ubuntu package for it. I think the right thing for us is to look into patching buildout so that it can get its setup_requires from its download cache.
Cheers, -Barry
-
Barry Warsaw -
P.J. Eby -
Zooko Wilcox-O'Hearn