setuptools 0.6c7 and zc.buildout: find-links egg fragment issue

Hello, If I create a buildout that uses subversion links to get some eggs, with a download cache, like this (I can provide a test case): [buildout] find-links = http://svn.example.com/my.package/trunk#egg=my.package http://svn.example.com/my.package2/trunk#egg=my.package2 egg = my.package my.package2 download-cache = /Users/tziade/.buildout/downloads/ It will fail because setuptools will try to extract the egg name from the url, using this code in package_index._download_url: name = filter(None,urlparse.urlparse(url)[2].split('/')) The name will then be "trunk" and the method will fail to work properly on the next download (my.package2) because the "trunk" folder will already be there: IOError: [Errno 21] Is a directory: '/Users/tziade/.buildout/downloads/trunk' That's because the function should extract the egg name from the egg fragment. If I add this code there: if '#egg=' in url: name = url.split('#=egg')[-1].strip() else: ... existing code to get the name.. It will work fine, because it will use unique egg names for folders. I would like to suggest changing the code this way. ++ Tarek -- View this message in context: http://www.nabble.com/setuptools-0.6c7-and-zc.buildout%3A-find-links-egg-fra... Sent from the Python - distutils-sig mailing list archive at Nabble.com.

FWIW, I find this feature to be baroque. I have very little interest in supporting it in buildout. (I wouldn't go out of my way to break it either.) I'd prefer to explore other ways to deal with the underlying use case in the context of buildout. I've tried to ignore this issue broadly, but I'm willing to work on an alternate solution because just seeing the discussion go by is too painful to keep ignoring. Jim On Feb 7, 2008, at 3:42 PM, Tarek Ziadé wrote:
Hello,
If I create a buildout that uses subversion links to get some eggs, with a download cache, like this (I can provide a test case):
[buildout]
find-links = http://svn.example.com/my.package/trunk#egg=my.package http://svn.example.com/my.package2/trunk#egg=my.package2
egg = my.package my.package2
download-cache = /Users/tziade/.buildout/downloads/
It will fail because setuptools will try to extract the egg name from the url, using this code in package_index._download_url:
name = filter(None,urlparse.urlparse(url)[2].split('/'))
The name will then be "trunk" and the method will fail to work properly on the next download (my.package2) because the "trunk" folder will already be there:
IOError: [Errno 21] Is a directory: '/Users/tziade/.buildout/downloads/trunk'
That's because the function should extract the egg name from the egg fragment. If I add this code there:
if '#egg=' in url: name = url.split('#=egg')[-1].strip() else: ... existing code to get the name..
It will work fine, because it will use unique egg names for folders. I would like to suggest changing the code this way.
++ Tarek
-- View this message in context: http://www.nabble.com/setuptools-0.6c7-and-zc.buildout%3A-find-links-egg-fra... Sent from the Python - distutils-sig mailing list archive at Nabble.com.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-- Jim Fulton Zope Corporation

Jim Fulton wrote:
FWIW, I find this feature to be baroque. I have very little interest in supporting it in buildout. (I wouldn't go out of my way to break it either.) I'd prefer to explore other ways to deal with the underlying use case in the context of buildout. I've tried to ignore this issue broadly, but I'm willing to work on an alternate solution because just seeing the discussion go by is too painful to keep ignoring.
Ok. The use case is : we are working on a bunch of private packages that do not have a public distribution. So we start a buildout like this: [buildout] develop = ../../packages/my.package ../../packages/my.package2 eggs = my.package my.package2 Then we create for each package a tag on the svn. And we want to distribute our buildout. so we change the buildout this way: [buildout] find-links = http://private/packages/my.package/tags/0.1#egg=my.package http://private/packages/my.package2/tags/0.1#egg=my.package2 extensions = lovely.buildouthttp download-cache = downloads eggs = my.package my.package2 ..to build a buildout that will have those eggs built within the folder. Then we add "install-from-cache" and "offline" to provide a installable tarball. Ok, we could release some eggs in some private url to have the proper find-links, but the #egg is quite convenient to avoid this extra step. So if we can find a way to automate this.. 2/ the develop problem Another point that would be great to consider: setting up the develop section can be painful when you share a project among many developers, because they need to svn checkout many packages and make sure they have the same folder structure. So could you consider this feature here : http://pypi.python.org/pypi/gp.svndevelop/0.1 ? We created this to be able to automate the set of develop on developers boxes. I think buildout misses this. 3/ the boostrap.py problem If we want to have a full offline mode (not using your rpm system) with a buildout directory, we need boostrap.py to avoid trying to get setuptools with an openurl call. So we have created a special boostrap.py file that first looks in the eggs folder, to be sure to have a 100% offline mode https://ingeniweb.svn.sourceforge.net/svnroot/ingeniweb/IngeniSkel/trunk/ing... Could you consider such a feature as well ? We work a *lot* with zc.buildout here, and those three points are the three things that are missing, in the way we use it. I can spare some time to work on some branches. ++ Tarek -- View this message in context: http://www.nabble.com/setuptools-0.6c7-and-zc.buildout%3A-find-links-egg-fra... Sent from the Python - distutils-sig mailing list archive at Nabble.com.

Tarek Ziadé wrote:
We work a *lot* with zc.buildout here, and those three points are the three things that are missing, in the way we use it. I can spare some time to work on some branches.
Oups I forgot one point : being able to tell that a given part in the buildout will be run only under a given platform win32. ++ tarek -- View this message in context: http://www.nabble.com/setuptools-0.6c7-and-zc.buildout%3A-find-links-egg-fra... Sent from the Python - distutils-sig mailing list archive at Nabble.com.

On Feb 7, 2008, at 4:14 PM, Tarek Ziadé wrote:
Jim Fulton wrote:
FWIW, I find this feature to be baroque. I have very little interest in supporting it in buildout. (I wouldn't go out of my way to break it either.) I'd prefer to explore other ways to deal with the underlying use case in the context of buildout. I've tried to ignore this issue broadly, but I'm willing to work on an alternate solution because just seeing the discussion go by is too painful to keep ignoring.
Ok. The use case is : we are working on a bunch of private packages that do not have a public distribution.
Have you considered a private repository?
So we start a buildout like this:
[buildout]
develop = ../../packages/my.package ../../packages/my.package2
eggs = my.package my.package2
Then we create for each package a tag on the svn. And we want to distribute our buildout. so we change the buildout this way:
Is your intention to distribute it for production deployment? Or for development?
[buildout]
find-links = http://private/packages/my.package/tags/0.1#egg=my.package http://private/packages/my.package2/tags/0.1#egg=my.package2
extensions = lovely.buildouthttp
download-cache = downloads
eggs = my.package my.package2
..to build a buildout that will have those eggs built within the folder. Then we add "install-from-cache" and "offline" to provide a installable tarball.
Have you looked at zc.sourcerelease?
Ok, we could release some eggs in some private url to have the proper find-links, but the #egg is quite convenient to avoid this extra step.
Just to avoid making a distro? You're already making an tag. Why not just use externals?
So if we can find a way to automate this..
I'm still not clear on what "this" is. Let me see, you want to be able to distribute software that includes develop eggs from svn. Is that right?
2/ the develop problem
Another point that would be great to consider: setting up the develop section can be painful when you share a project among many developers, because they need to svn checkout many packages and make sure they have the same folder structure.
Unless you use externals.
So could you consider this feature here : http://pypi.python.org/pypi/gp.svndevelop/0.1 ? We created this to be able to automate the set of develop on developers boxes. I think buildout misses this.
Why not use externals?
3/ the boostrap.py problem
If we want to have a full offline mode (not using your rpm system)
You mean without zc.sourcerelease?
with a buildout directory, we need boostrap.py to avoid trying to get setuptools with an openurl call.
So we have created a special boostrap.py file that first looks in the eggs folder, to be sure to have a 100% offline mode
https://ingeniweb.svn.sourceforge.net/svnroot/ingeniweb/IngeniSkel/trunk/ing...
Could you consider such a feature as well ?
Sure. I'm unclear on why you don't want to use zc.sourcerelease?
We work a *lot* with zc.buildout here, and those three points are the three things that are missing, in the way we use it. I can spare some time to work on some branches.
Cool.
Oups I forgot one point : being able to tell that a given part in the buildout will be run only under a given platform win32.
Right, so for other folks benefit, the idea is to be able to say something like: [foo platform=win32] and have the options defined there provide foo options only on win32. I definitely plan to do this. Jim -- Jim Fulton Zope Corporation

Jim Fulton wrote:
Ok. The use case is : we are working on a bunch of private packages that do not have a public distribution.
Have you considered a private repository?
It is in a private repository. Jim Fulton wrote:
Then we create for each package a tag on the svn. And we want to distribute our buildout. so we change the buildout this way:
Is your intention to distribute it for production deployment? Or for development?
For production deployement Jim Fulton wrote:
Then we add "install-from-cache" and "offline" to provide a installable tarball.
Have you looked at zc.sourcerelease?
We have quite a similar script that builds a tarball, we created before you pointed me to zc.sourcerelease. I guess your script handles the bootstrap part differently if you didn't need to change it. So I will try that. But we need to keep our script because it has more advanced features iirc for our upgrade needs (like making diffs between two buildouts) Jim Fulton wrote:
Ok, we could release some eggs in some private url to have the proper find-links, but the #egg is quite convenient to avoid this extra step.
Just to avoid making a distro? You're already making an tag.
Why not just use externals?
We are juste tagging. Making a distro is an extra step. Setting up a bundle is also an extra step. Jim Fulton wrote:
So if we can find a way to automate this..
I'm still not clear on what "this" is. Let me see, you want to be able to distribute software that includes develop eggs from svn. Is that right?
No, I want to be able to distribute software that includes eggs from svn. Jim Fulton wrote:
2/ the develop problem
Another point that would be great to consider: setting up the develop section can be painful when you share a project among many developers, because they need to svn checkout many packages and make sure they have the same folder structure.
Unless you use externals.
Yup. but that's an extra step, because you need to build a bundle, then make sure the buildout and the bundle are always synchronized. The buildout was a way to get rid of bundles imho. If I need to work with a bundle, it's a step back in my opinion. If we could point an egg in a svn like we do on a file system: develop = /path/in/the/system/my.egg svn://path/in/the/svn/my.egg2/trunk https://path/in/the/svn/my.egg3/trunk That would avoid maintaining a bundle Jim Fulton wrote:
3/ the boostrap.py problem If we want to have a full offline mode (not using your rpm system)
You mean without zc.sourcerelease?
I guess.. Jim Fulton wrote:
Oups I forgot one point : being able to tell that a given part in the buildout will be run only under a given platform win32.
Right, so for other folks benefit, the idea is to be able to say something like:
[foo platform=win32]
and have the options defined there provide foo options only on win32.
I definitely plan to do this.
Great ! ++ Tarek -- View this message in context: http://www.nabble.com/setuptools-0.6c7-and-zc.buildout%3A-find-links-egg-fra... Sent from the Python - distutils-sig mailing list archive at Nabble.com.
participants (2)
-
Jim Fulton
-
Tarek Ziadé