zc.buildout fails to use system-installed dep?
Hello, there's a problem with zc.buildout that I can reproduce very easily on Ubuntu 10.10 (64 bit) it seems that zc.buildout does not download the dependency from pypi if the very same version is already available in the host system, but then for some reason does not add it to sys.path and/or is unable to use it for some reason; zope.testrunner hence fails on importing zope.interface python-zope.interface is installed at version 3.6.1, and zope.testrunner seems to use that very same dependency. Any workaround? I'll open an issue on launchpad in the weekend. Do this in order to reproduce the problem: # start cd /tmp mkdir -p MyProj/myproj cd MyProj cat <<EOM > buildout.cfg [buildout] parts = test develop = myproj [test] recipe = zc.recipe.testrunner eggs = myproj EOM cat <<EOM > myproj/setup.py from setuptools import setup setup( name="myproj", version="1.0.0", description="myproj" ) EOM wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py python bootstrap.py --distribute && bin/buildout -vvv bin/test #end output from buildout run: alan@afra:/tmp/MyProj$ python bootstrap.py --distribute && bin/buildout Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.14.tar.gz Extracting in /tmp/tmpUAAumj Now working in /tmp/tmpUAAumj/distribute-0.6.14 Building a Distribute egg in /tmp/tmphpzBsD /tmp/tmphpzBsD/distribute-0.6.14-py2.6.egg install_dir /tmp/tmphpzBsD Creating directory '/tmp/MyProj/bin'. Creating directory '/tmp/MyProj/parts'. Creating directory '/tmp/MyProj/eggs'. Creating directory '/tmp/MyProj/develop-eggs'. Generated script '/tmp/MyProj/bin/buildout'. Upgraded: distribute version 0.6.14; restarting. Develop: '/tmp/MyProj/myproj' install_dir /tmp/MyProj/develop-eggs/tmpoKKOZAbuild Getting distribution for 'zc.recipe.testrunner'. install_dir /tmp/MyProj/eggs/tmpQvIfIg zip_safe flag not set; analyzing archive contents... Got zc.recipe.testrunner 1.4.0. Getting distribution for 'z3c.recipe.scripts>=1.0.0'. install_dir /tmp/MyProj/eggs/tmpQPz2RH Got z3c.recipe.scripts 1.0.1. Getting distribution for 'zope.testrunner'. install_dir /tmp/MyProj/eggs/tmpGvi6JB package init file 'src/zope/testrunner/testrunner-ex/__init__.py' not found (or not a regular file) Creating missing __init__.py for zope.testrunner.testrunner-ex Got zope.testrunner 4.0.0. Getting distribution for 'zc.recipe.egg>=1.3.0'. install_dir /tmp/MyProj/eggs/tmp_pJ4gb Got zc.recipe.egg 1.3.2. Getting distribution for 'zope.exceptions'. install_dir /tmp/MyProj/eggs/tmpxRSjc1 Got zope.exceptions 3.6.1. Installing test. Generated script '/tmp/MyProj/bin/test'. alan@afra:/tmp/MyProj$ bin/test Traceback (most recent call last): File "bin/test", line 20, in <module> import zope.testrunner File "/tmp/MyProj/eggs/zope.testrunner-4.0.0-py2.6.egg/zope/testrunner/__init__.py", line 21, in <module> import zope.testrunner.interfaces File "/tmp/MyProj/eggs/zope.testrunner-4.0.0-py2.6.egg/zope/testrunner/interfaces.py", line 21, in <module> import zope.interface ImportError: No module named interface -- Alan Franzoni -- contact me at public@[mysurname].eu
Hi, On Fri, Dec 3, 2010 at 18:28, Alan Franzoni <mailing@franzoni.eu> wrote:
Hello, there's a problem with zc.buildout that I can reproduce very easily on Ubuntu 10.10 (64 bit)
it seems that zc.buildout does not download the dependency from pypi if the very same version is already available in the host system, but then for some reason does not add it to sys.path and/or is unable to use it for some reason; zope.testrunner hence fails on importing zope.interface
python-zope.interface is installed at version 3.6.1, and zope.testrunner seems to use that very same dependency.
Any workaround? I'll open an issue on launchpad in the weekend.
Do this in order to reproduce the problem:
I ran your script, and the result ended with: Total: 0 tests, 0 failures, 0 errors in 0.000 seconds. I also have zope.interface 3.6.1 on my system. Gentoo, Python 2.7.1+. $ eix zope.interface [D] net-zope/zope-interface Available versions: 3.5.1 3.5.2 (~)3.5.3 Installed versions: 3.6.1(06:23:08 10/23/10) Homepage: http://pypi.python.org/pypi/zope.interface Description: Interfaces for Python Perhaps try adding this to the buildout config: include-site-packages = true allowed-eggs-from-site-packages = zope.interface Attila
I had only linux to test it with, and I could reproduce the bug on two distinct Ubuntu 10.10 64 bit workstations. It doesn't happen on Ubuntu 10.04, but that distro is got zope.interface 3.5.something and I can see zope.interface gets downloaded again. It doesn't happen on mac os x with python 2.7 either, even though zope.interface was installed through easy_install. I forgot to specify that Ubuntu 10.10 is got Python 2.6.5. I suspect it's something ubuntu-specific, maybe related to the fact that it's not an egg-installed package, but a .deb installed one - some interference between the two install systems is not unlikely. Just some questions for you, Attila: - was your zope.interface installed through portage or via easy_install/pip? - can you see zope.interface 3.6.1 being downloaded during the buildout launch phase? -- Alan Franzoni -- contact me at public@[mysurname].eu
Hi, On Fri, Dec 3, 2010 at 20:35, Alan Franzoni <mailing@franzoni.eu> wrote:
- was your zope.interface installed through portage or via easy_install/pip? Portage.
This is what I get from IPython though: In [1]: import zope In [2]: zope Out[2]: <module 'zope' (built-in)> In [3]: import zope.interface In [4]: zope.interface Out[4]: <module 'zope.interface' from '/usr/lib64/python2.7/site-packages/zope/interface/__init__.pyc'> aatiis@aiur ~ $ ls /usr/lib/python2.7/site-packages/zope* /usr/lib/python2.7/site-packages/zope.interface-3.6.1-py2.7-nspkg.pth /usr/lib/python2.7/site-packages/zope: interface /usr/lib/python2.7/site-packages/zope.interface-3.6.1-py2.7.egg-info: PKG-INFO SOURCES.txt dependency_links.txt namespace_packages.txt not-zip-safe requires.txt top_level.txt
- can you see zope.interface 3.6.1 being downloaded during the buildout launch phase?
Nope, I see something like "Egg from site-packages: zope.interface" I guess Gentoo installs a valid egg using portage, and buildout can recognize it. Attila
I've verified this happens all the times on Ubuntu 10.10, both 32 and 64 bit. I don't know if it's Ubuntu's or zc.buildout's fault; I've opened a ticket. -- Alan Franzoni -- contact me at public@[mysurname].eu
For the records, or if anybody hits this problem before it's fixed; the only way to cope with this that I found is to add include-site-packages = false in the configuration. This forces the redownload of the egg and everything works fine. -- Alan Franzoni -- contact me at public@[mysurname].eu
On 12/05/2010 11:14 PM, Alan Franzoni wrote:
I've verified this happens all the times on Ubuntu 10.10, both 32 and 64 bit. I don't know if it's Ubuntu's or zc.buildout's fault; I've opened a ticket.
I have the same problem on ubuntu. I never managed to find out what the real reason is, but my guess is that it's ubuntu's way of installing python packages for multiple python versions. You have a /usr/lib/python2.6/dist-packages/ directory, but also a /usr/lib/pyshared/python2.6/, for instance. Something somewhere is doing some funny sys.path mangling, probably ubuntu. The end result is that I still haven't gotten buildout 1.5.x to work on ubuntu with the system packages that I need. I'm still at 1.4.4 with the custom 1.4-x-only bootstrap, in combination with osc.recipe.sysegg I hope to have some debugging-time in January, as it is pretty irritating not to be able to use buildout 1.5.x ;-) Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Collega's gezocht! Django/python vacature in Utrecht: http://tinyurl.com/35v34f9
On Wed, Dec 8, 2010 at 12:21 PM, Reinout van Rees <reinout@vanrees.org> wrote:
I have the same problem on ubuntu. I never managed to find out what the real reason is, but my guess is that it's ubuntu's way of installing python packages for multiple python versions.
I have filed a ticket on zc.buildout bugtracker, you could use the "this affects me" feature. https://bugs.launchpad.net/zc.buildout/+bug/685638
You have a /usr/lib/python2.6/dist-packages/ directory, but also a /usr/lib/pyshared/python2.6/, for instance. Something somewhere is doing some funny sys.path mangling, probably ubuntu.
Yes, I suppose it's a kind of interference. I don't really know who's causing the problems, I think we'd need some feedback from both zc.buildout and ubuntu developers, but I can see Ubuntu still packages 1.4.3 even in Natty - there might be a reason for that. -- Alan Franzoni -- contact me at public@[mysurname].eu
On Wed, Dec 08, 2010 at 12:21:55PM +0100, Reinout van Rees wrote:
On 12/05/2010 11:14 PM, Alan Franzoni wrote:
I've verified this happens all the times on Ubuntu 10.10, both 32 and 64 bit. I don't know if it's Ubuntu's or zc.buildout's fault; I've opened a ticket.
I have the same problem on ubuntu. I never managed to find out what the real reason is, but my guess is that it's ubuntu's way of installing python packages for multiple python versions. You have a /usr/lib/python2.6/dist-packages/ directory,
which is on sys.path
but also a /usr/lib/pyshared/python2.6/,
which isn't. But then there's /usr/lib/pymodules/python2.6, which is on sys.path, and I've no idea what it's about. dist-packages/python-support.pth adds this one.
for instance. Something somewhere is doing some funny sys.path mangling, probably ubuntu.
Debian and Ubuntu use dist-packages instead of site-packages to avoid conflicts with manual installations (I'm not sure if they mean manual as in sudo easy_install foo, or if they mean manual as in self-compiled Pythons. Probably the latter.). If zc.buildout assumes it can find system packages in site-packages instead of dist-packages, then it would fail. I've no idea if that's actually the case. /usr/share/doc/python/python-policy.txt.gz has more information about Debian and Ubuntu specifics.
The end result is that I still haven't gotten buildout 1.5.x to work on ubuntu with the system packages that I need. I'm still at 1.4.4 with the custom 1.4-x-only bootstrap, in combination with osc.recipe.sysegg
IIRC you can use the new bootstrap and ask it to get a specific version of zc.buildout. I never needed recipes to use system Python packages in 1.4 buildouts, but that was before Python 2.6 and dist-packages -- Debian's Python 2.5 still uses site-packages.
I hope to have some debugging-time in January, as it is pretty irritating not to be able to use buildout 1.5.x ;-)
Marius Gedminas -- "... one of the main causes of the fall of the Roman Empire was that, lacking zero, they had no way to indicate successful termination of their C programs." -- Robert Firth
To start with, the Debian/Ubuntu arrangement is made even more confusing for there being three competing "build systems" for Python. There are the traditional python-central and python-support systems, which most packages these days use. python-support is probably more popular than python-central. Both however are being obsoleted for the new goodness, dh_python2. The biggest benefit to dh_python2 is that the symlinks are created at build-time and appear in the package, instead of being created at install-time and managed by what can be flaky post-scripts. The conversion process will begin happening in earnest once Debian squeeze is released. Why do we have symlinks in the first place? It's because Debian and Ubuntu support multiple active versions of Python at the same time. Once Python 2 is killed off and we're all using >= Python 3.2, we can get rid of even this cruft. That's the major reason why I worked on PEPs 3147 and 3149. On Dec 08, 2010, at 06:32 PM, Marius Gedminas wrote:
On Wed, Dec 08, 2010 at 12:21:55PM +0100, Reinout van Rees wrote:
On 12/05/2010 11:14 PM, Alan Franzoni wrote:
I've verified this happens all the times on Ubuntu 10.10, both 32 and 64 bit. I don't know if it's Ubuntu's or zc.buildout's fault; I've opened a ticket.
I have the same problem on ubuntu. I never managed to find out what the real reason is, but my guess is that it's ubuntu's way of installing python packages for multiple python versions. You have a /usr/lib/python2.6/dist-packages/ directory,
which is on sys.path
but also a /usr/lib/pyshared/python2.6/,
which isn't.
But then there's /usr/lib/pymodules/python2.6, which is on sys.path, and I've no idea what it's about. dist-packages/python-support.pth adds this one.
All roads point to pyshared, which is where the files are *actually* installed. pymodules and dist-packages point into pyshared and are there to handle the multiple-installed-python arrangement. I think the difference comes about depending on whether python-central or python-support is being used.
for instance. Something somewhere is doing some funny sys.path mangling, probably ubuntu.
Debian and Ubuntu use dist-packages instead of site-packages to avoid conflicts with manual installations (I'm not sure if they mean manual as in sudo easy_install foo, or if they mean manual as in self-compiled Pythons. Probably the latter.).
Yes. A from-source build and install of Python would use /usr/local/lib/pythonX.Y/site-packages, but Debian's interpretation of the FHS puts that directory under the domain of the system administrator. Which means, it was possible to install third party packages into a from-source build of Python (with default configure options and 'make altinstall') and break your system Python, or vice versa. dist-packages is a compromise to the decade's tradition of Python source default locations. (Arguably, a from-source build of Python with the default options should install into /opt.)
If zc.buildout assumes it can find system packages in site-packages instead of dist-packages, then it would fail. I've no idea if that's actually the case.
I don't remember specifically for zc.buildout, but distribute for example has been modified to use dist-packages.
/usr/share/doc/python/python-policy.txt.gz has more information about Debian and Ubuntu specifics.
The end result is that I still haven't gotten buildout 1.5.x to work on ubuntu with the system packages that I need. I'm still at 1.4.4 with the custom 1.4-x-only bootstrap, in combination with osc.recipe.sysegg
IIRC you can use the new bootstrap and ask it to get a specific version of zc.buildout. I never needed recipes to use system Python packages in 1.4 buildouts, but that was before Python 2.6 and dist-packages -- Debian's Python 2.5 still uses site-packages.
I'm using zc.buildout 1.5.2 on Ubuntu 10.10 for Mailman 3. There is a known bug related to namespace packages, which force you to use include-site-packages=false in your buildout.cfg. Here's the bug report: https://bugs.launchpad.net/zc.buildout/+bug/659231 Gary is assigned to the bug but there is as yet no resolution. -Barry
On Wed, Dec 8, 2010 at 8:10 PM, Barry Warsaw <barry@python.org> wrote:
Why do we have symlinks in the first place? It's because Debian and Ubuntu support multiple active versions of Python at the same time. Once Python 2 is killed off and we're all using >= Python 3.2, we can get rid of even this cruft. That's the major reason why I worked on PEPs 3147 and 3149.
It might be a long time indeed. Python 2.7 will probably keep up for several years.
I'm using zc.buildout 1.5.2 on Ubuntu 10.10 for Mailman 3. There is a known bug related to namespace packages, which force you to use include-site-packages=false in your buildout.cfg. Here's the bug report:
https://bugs.launchpad.net/zc.buildout/+bug/659231
Gary is assigned to the bug but there is as yet no resolution.
It's the very same error and the same workaround is suggested, but I don't see enough proof that the reason behind such error lies within namespace_packages. I'll make some tests in the next days and I'll check whether it's the case. -- Alan Franzoni -- contact me at public@[mysurname].eu
On Dec 09, 2010, at 12:35 AM, Alan Franzoni wrote:
On Wed, Dec 8, 2010 at 8:10 PM, Barry Warsaw <barry@python.org> wrote:
Why do we have symlinks in the first place? It's because Debian and Ubuntu support multiple active versions of Python at the same time. Once Python 2 is killed off and we're all using >= Python 3.2, we can get rid of even this cruft. That's the major reason why I worked on PEPs 3147 and 3149.
It might be a long time indeed. Python 2.7 will probably keep up for several years.
Indeed. I'm glad I'm not the 2.7 release manager. :)
I'm using zc.buildout 1.5.2 on Ubuntu 10.10 for Mailman 3. There is a known bug related to namespace packages, which force you to use include-site-packages=false in your buildout.cfg. Here's the bug report:
https://bugs.launchpad.net/zc.buildout/+bug/659231
Gary is assigned to the bug but there is as yet no resolution.
It's the very same error and the same workaround is suggested, but I don't see enough proof that the reason behind such error lies within namespace_packages. I'll make some tests in the next days and I'll check whether it's the case.
Please do. Cheers, -Barry
On Thu, Dec 9, 2010 at 12:39 AM, Barry Warsaw <barry@python.org> wrote: [cut] After taking a look, it seems that namespace packages are involved indeed, even though I'm not sure how yet, and there seems to be another related bug where you commented as well: https://bugs.launchpad.net/ubuntu/+source/python2.6/+bug/621348 I admit I'm not extremely expert at import matters, and I'm getting mad at understanding how many levels of indirection are involved when resolving a python import! I tinkered around a bit trying to check whether the behaviour could be reproduced for other modules. It seems it cannot! I've tried installing zc.lockfile from Ubuntu packages, then i added the "zc.dict" egg to my test recipe. I expected to be unable to import zc.lockfile, but that wasn't the case. nspkg.pth files exists both for zc.lockfile and zope.interface, but while the dist-packages/zope dir has got a __init__.py file with declare_namespace(__name__) in it, the dist-packages/zc dir hasn't. I removed the __init__.py file in dist-packages/zope and now I'm able to run my script without a flaw. I must admit I don't fully understand nspkg.pth files (where can I find more info on them? Who's creating them, setuptools/distribute?), but their usage and the declare_namespace method seems mutually exclusive by the way; was that confusing zc.buildout, I suppose? Should I open a ticket on ubuntu zope.interface package? -- Alan Franzoni -- contact me at public@[mysurname].eu
On Thu, Dec 9, 2010 at 11:25 PM, Alan Franzoni <mailing@franzoni.eu> wrote: [cut] It appears that the zope.interface mantainer from Debian manually copies the unnecessary - and confusing - __init__.py files: http://svn.debian.org/viewsvn/pkg-zope/zope.interface/trunk/debian/rules?revision=2074&view=markup It seems the only package in the python-zope.* hierarchy doing that; I don't know why. I have opened a ticket in Ubuntu: https://bugs.launchpad.net/ubuntu/+source/zope.interface/+bug/688335 I'm unable to do the same in Debian although it's probably there, because they seem to have no web interface and I don't have a Debian machine available right now. If anybody could check it and do so it would be great. -- Alan Franzoni -- contact me at public@[mysurname].eu
On Fri, Dec 10, 2010 at 12:17:02AM +0100, Alan Franzoni wrote:
On Thu, Dec 9, 2010 at 11:25 PM, Alan Franzoni <mailing@franzoni.eu> wrote: [cut]
It appears that the zope.interface mantainer from Debian manually copies the unnecessary - and confusing - __init__.py files:
http://svn.debian.org/viewsvn/pkg-zope/zope.interface/trunk/debian/rules?revision=2074&view=markup
It seems the only package in the python-zope.* hierarchy doing that; I don't know why.
I have opened a ticket in Ubuntu:
https://bugs.launchpad.net/ubuntu/+source/zope.interface/+bug/688335
I'm unable to do the same in Debian although it's probably there, because they seem to have no web interface and I don't have a Debian machine available right now. If anybody could check it and do so it would be great.
Please don't, this has been reported various times already. Removing the zope/__init__.py causes breakage in other places, as I detail in my response to your bug. I would guess that the real bug is at a deeper level. -- Brian Sutherland
On Fri, Dec 10, 2010 at 9:36 AM, Brian Sutherland <brian@vanguardistas.net> wrote:
Please don't, this has been reported various times already.
Removing the zope/__init__.py causes breakage in other places, as I detail in my response to your bug.
Sigh, I guess we'd need a Ubuntu/Debian python packaging FAQ? If it has been reported many times but keeps popping up, it's probably unpredictable enough. It took me 3 hours just on yesterday evening to find a workaround to that, and I'd have gladly spared that :-( -- Alan Franzoni -- contact me at public@[mysurname].eu
On Fri, Dec 10, 2010 at 10:34:57AM +0100, Alan Franzoni wrote:
On Fri, Dec 10, 2010 at 9:36 AM, Brian Sutherland <brian@vanguardistas.net> wrote:
Please don't, this has been reported various times already.
Removing the zope/__init__.py causes breakage in other places, as I detail in my response to your bug.
Sigh, I guess we'd need a Ubuntu/Debian python packaging FAQ?
If it has been reported many times but keeps popping up, it's probably unpredictable enough. It took me 3 hours just on yesterday evening to find a workaround to that, and I'd have gladly spared that :-(
Sorry, if I'd realized sooner that your issues with buildout were related, I would have spoken up sooner. I've also lost many hours of my life due to this one :( I'll mark your launchpad bug for now as Wontfix and hope that the deeper bug gets resolved sometime. Unfortunately even figuring out what that is (or what causes it) is beyond my ken. -- Brian Sutherland
On Dec 10, 2010, at 09:36 AM, Brian Sutherland wrote:
Please don't, this has been reported various times already.
Removing the zope/__init__.py causes breakage in other places, as I detail in my response to your bug.
I would guess that the real bug is at a deeper level.
Just a couple of things: The way zope.interface "owns" the zope namespace is not correct. Ideally, there would be a separate binary package that owns zope/__init__.py and on which all other zope subpackages depends. This could of course be built from the zope.interface source package. Second, we're going to make a big push after Squeeze is released to convert packaging to use dh_python2, the new goodness in Debian Python packaging. This should make things more consistent, and, while I have not yet tracked down specific details yet, it should make handling namespace packages much more robust. For example, with my flufl packages, there is no binary package owning flufl/__init__.py. That file does not show up in the binary package and yet it still gets created properly when any of the subpackages get installed. Sadly PEP 382 was not complete in time for Python 3.2. -Barry
On Fri, Dec 10, 2010 at 07:17:16AM -0500, Barry Warsaw wrote:
On Dec 10, 2010, at 09:36 AM, Brian Sutherland wrote:
Please don't, this has been reported various times already.
Removing the zope/__init__.py causes breakage in other places, as I detail in my response to your bug.
I would guess that the real bug is at a deeper level.
Just a couple of things:
The way zope.interface "owns" the zope namespace is not correct. Ideally, there would be a separate binary package that owns zope/__init__.py and on which all other zope subpackages depends. This could of course be built from the zope.interface source package.
I originally had it that way, but was strongly advised to change it to the current method to be able to pass the Debian FTP Master gauntlet. The current method using dpkg-divert is not too bad, more packages than python-zope.interface could include that file as well. So you don't force installation of python-zope.interface. It also uses standard dpkg functionality, which is a robustness bonus.
Second, we're going to make a big push after Squeeze is released to convert packaging to use dh_python2, the new goodness in Debian Python packaging.
I've had a brief look at it already and will have a much deeper look once squeeze is released. AFAIKR the only feature it was missing to completely cover my usecase was good handling of setuptools extras. I'm hoping to be able to completely replace the custom tools we use in python-zope.* packages with it at some point.
This should make things more consistent, and, while I have not yet tracked down specific details yet, it should make handling namespace packages much more robust. For example, with my flufl packages, there is no binary package owning flufl/__init__.py. That file does not show up in the binary package and yet it still gets created properly when any of the subpackages get installed.
Any idea as to the mechanism? What about 2nd level namespace packages (horror: zope.app.foo)?
Sadly PEP 382 was not complete in time for Python 3.2.
Yeah, there are a lot of packaging related PEPs coming out lately. It's really great to see attention being paid to these dusty corners:) -- Brian Sutherland
On Dec 10, 2010, at 01:42 PM, Brian Sutherland wrote:
On Fri, Dec 10, 2010 at 07:17:16AM -0500, Barry Warsaw wrote:
The way zope.interface "owns" the zope namespace is not correct. Ideally, there would be a separate binary package that owns zope/__init__.py and on which all other zope subpackages depends. This could of course be built from the zope.interface source package.
I originally had it that way, but was strongly advised to change it to the current method to be able to pass the Debian FTP Master gauntlet.
Just goes to illustrate the myth of TOOWTDI. :) AFAICT, from my discussions with various folks on debian-python, it should now[*] be done with the separate binary package. [*] At least until dh_python2, if we can ensure it always DTRT.
The current method using dpkg-divert is not too bad, more packages than python-zope.interface could include that file as well. So you don't force installation of python-zope.interface.
Doing the diversion means it's harder for someone to explicitly determine which package owns the file. Better (I think!) to have none of them own the file.
It also uses standard dpkg functionality, which is a robustness bonus.
True.
Second, we're going to make a big push after Squeeze is released to convert packaging to use dh_python2, the new goodness in Debian Python packaging.
I've had a brief look at it already and will have a much deeper look once squeeze is released. AFAIKR the only feature it was missing to completely cover my usecase was good handling of setuptools extras.
Can you be more specific? Before I got swamped with the Python 2.7 transition (it will be the default in Ubuntu 11.04), I began looking at dh_python2, adding unit tests, etc. I do plan to get back to it once we have the archive more happily on Python 2.7[*]. [*] https://bugs.launchpad.net/ubuntu/+bugs?field.tag=python27
I'm hoping to be able to completely replace the custom tools we use in python-zope.* packages with it at some point.
+1. We hope to get rid of python-support and python-central.
This should make things more consistent, and, while I have not yet tracked down specific details yet, it should make handling namespace packages much more robust. For example, with my flufl packages, there is no binary package owning flufl/__init__.py. That file does not show up in the binary package and yet it still gets created properly when any of the subpackages get installed.
Any idea as to the mechanism?
Hints, but nothing definite. I'm not sure which part of the tool chain is suppressing the neamespace package's __init__.py, and which part is laying it down when the package gets installed. It *is* nice that the packager doesn't have to worry about it though. I don't think you should have to do the tricks zope.interface is doing, or even do the extra binary package. IOW, it should Just Work.
What about 2nd level namespace packages (horror: zope.app.foo)?
Haven't tried that yet.
Sadly PEP 382 was not complete in time for Python 3.2.
Yeah, there are a lot of packaging related PEPs coming out lately. It's really great to see attention being paid to these dusty corners:)
Indeed! I think we all owe Tarek and the other folks an unlimited supply of beers at the next Pycon. :) -Barry
On Fri, Dec 10, 2010 at 2:19 PM, Barry Warsaw <barry@python.org> wrote:
On Dec 10, 2010, at 01:42 PM, Brian Sutherland wrote:
[cut] A new, interesting twist! A zc.buildout developer has answered to my original ticket: https://bugs.launchpad.net/zc.buildout/+bug/685638 and suggested I took a look at another ticket he reported for Ubuntu: https://bugs.launchpad.net/ubuntu/+source/distribute/+bug/576434 Following the instruction of the latter I was able to use zope.interface without an issue, even though __init__.py it's inside it. By the way: things are getting way, way too complicated. I can imagine nobody has really a good grasp on what's going on "under the hood". I'm looking forward to help with packaging tools before it's too late :-) -- Alan Franzoni -- contact me at public@[mysurname].eu
On Fri, Dec 10, 2010 at 08:19:32AM -0500, Barry Warsaw wrote:
On Dec 10, 2010, at 01:42 PM, Brian Sutherland wrote:
On Fri, Dec 10, 2010 at 07:17:16AM -0500, Barry Warsaw wrote:
The way zope.interface "owns" the zope namespace is not correct. Ideally, there would be a separate binary package that owns zope/__init__.py and on which all other zope subpackages depends. This could of course be built from the zope.interface source package.
I originally had it that way, but was strongly advised to change it to the current method to be able to pass the Debian FTP Master gauntlet.
Just goes to illustrate the myth of TOOWTDI. :) AFAICT, from my discussions with various folks on debian-python, it should now[*] be done with the separate binary package.
[*] At least until dh_python2, if we can ensure it always DTRT.
I'll just till dh_python2 then before changing anything ;) hmm, perhaps I should join debina-python again... <snip>
Second, we're going to make a big push after Squeeze is released to convert packaging to use dh_python2, the new goodness in Debian Python packaging.
I've had a brief look at it already and will have a much deeper look once squeeze is released. AFAIKR the only feature it was missing to completely cover my usecase was good handling of setuptools extras.
Can you be more specific? Before I got swamped with the Python 2.7 transition (it will be the default in Ubuntu 11.04), I began looking at dh_python2, adding unit tests, etc. I do plan to get back to it once we have the archive more happily on Python 2.7[*].
[*] https://bugs.launchpad.net/ubuntu/+bugs?field.tag=python27
I always use zope.component as an example because it has the worst case use of setuptools extras which a lot of other packages depend on. In 3.9.1 this is: extras_require = dict( hook = ['zope.hookable'], persistentregistry = ['ZODB3'], zcml = ['zope.configuration', 'zope.i18nmessageid', ], test = ['ZODB3', 'zope.testing', 'zope.hookable', 'zope.location', 'zope.proxy', 'zope.security', ], docs = ['z3c.recipe.sphinxdoc'], ), The way we treat these extras in the python-zope.* packages is as if they were extra binary packages with names like python-${distribution_name}-${extra_name}. So zope.component is actually 6 binary packages: python-zope.component python-zope.component-hook python-zope.component-persistentregistry python-zope.component-zcml python-zope.component-test python-zope.component-docs But, we can't make 6 binary packages or the FTP Masters will kill us. So we do a few things with extras: * Have Ignore them and their dependencies. * Provide the extra package from the main package and depend on it's dependencies. * Have their dependencies as "suggests" on the main package * Break them out into a binary metapackage. python-zope.component does all of these for it's extras as appropriate in each case. I wrote a plugin to debhelper 7 so this can be specified in an easy way using environment variables in the rules file: http://svn.debian.org/viewsvn/pkg-zope/zope.component/trunk/debian/rules?revision=2053&view=markup The code to do the work is in the python-van.pydeb package. One of the things I had hoped to do after squeeze was port the van.pydep tests to dh_python2 and then bother someone else to fix them. python-zope.security is also another tough case. In that case, Ubuntu needs to carry a diff against Debian to cope with the "untrustedpython" extra. <snip>
This should make things more consistent, and, while I have not yet tracked down specific details yet, it should make handling namespace packages much more robust. For example, with my flufl packages, there is no binary package owning flufl/__init__.py. That file does not show up in the binary package and yet it still gets created properly when any of the subpackages get installed.
Any idea as to the mechanism?
Hints, but nothing definite. I'm not sure which part of the tool chain is suppressing the neamespace package's __init__.py, and which part is laying it down when the package gets installed. It *is* nice that the packager doesn't have to worry about it though. I don't think you should have to do the tricks zope.interface is doing, or even do the extra binary package. IOW, it should Just Work.
+100 to just working :)
What about 2nd level namespace packages (horror: zope.app.foo)?
Haven't tried that yet.
Well, cross that bridge when we come to it...
Sadly PEP 382 was not complete in time for Python 3.2.
Yeah, there are a lot of packaging related PEPs coming out lately. It's really great to see attention being paid to these dusty corners:)
Indeed! I think we all owe Tarek and the other folks an unlimited supply of beers at the next Pycon. :)
Yep :) -- Brian Sutherland
On Fri, Dec 10, 2010 at 08:19:32AM -0500, Barry Warsaw wrote:
Yeah, there are a lot of packaging related PEPs coming out lately. It's really great to see attention being paid to these dusty corners:)
Indeed! I think we all owe Tarek and the other folks an unlimited supply of beers at the next Pycon. :)
Oh yes. And PJE too, for kickstarting the whole thing when he came up with setuptools. Marius Gedminas -- Anybody who doesn't cut his speed at the sight of a police car is probably parked.
On 10.12.2010 13:42, Brian Sutherland wrote:
The current method using dpkg-divert is not too bad, more packages than python-zope.interface could include that file as well. So you don't force installation of python-zope.interface.
It also uses standard dpkg functionality, which is a robustness bonus.
A diversion only works if not more than one package diverts the file. You would have to use alternatives to handle the file for more than two packages, although you would most likely point to instances of the same file. Matthias
participants (7)
-
Alan Franzoni -
Attila Oláh -
Barry Warsaw -
Brian Sutherland -
Marius Gedminas -
Matthias Klose -
Reinout van Rees