Appears bdist_rpm completely broken
Maybe I'm missing something, but I don't see how bdist_rpm works for anyone. I've tried 0.9.2, the 9/12 and 9/16 snapshots but I have a problem with the generated pycs.
First the symptom. I build 4Suite using distutils. The sdist, bdist and bdist_win packages work just fine, but when I move the generated RPM to another machine and try to use 4Suite:
[uogbuji@dynip-23 uogbuji]$ python Python 1.5.2 (#1, Jul 12 2000, 10:39:14) [GCC 2.96 20000702 (experimental)] on linux-i386 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
import sys sys.path
['', '/usr/lib/python1.5/', '/usr/lib/python1.5/plat-linux-i386', '/usr/lib/python1.5/lib-tk', '/usr/lib/python1.5/lib-dynload', '/usr/lib/python1.5/site-packages']
import xml.xslt
Traceback (innermost last): File "<stdin>", line 1, in ? File "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xslt/__init__.py", line 3, in ? File "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xpath/__init__.py", line 3, in ? ImportError: cannot import name XPathParserBase
The problem is the phantom "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages"...
[uogbuji@dynip-23 uogbuji]$ ls /var/tmp [uogbuji@dynip-23 uogbuji]$
Further investigation turns up the culprit: The generated pyc files from the RPM have the "/var/tmp/..." hard-coded in to their imports.
So am I doing something wrong, or is distutils broken for RPM build?
I'm not on the distutils list so ccs to me would be appreciated. Otherwise I can check the archives.
Thanks.
On Wed, Sep 20, 2000 at 12:06:23PM -0600, Uche Ogbuji wrote:
Maybe I'm missing something, but I don't see how bdist_rpm works for anyone. I've tried 0.9.2, the 9/12 and 9/16 snapshots but I have a problem with the generated pycs.
It generated ZODB & ExtensionClass RPMs that worked fine, and those modules both involve mixed Python and C module, which is about as complicated as you can get.
line 3, in ? ImportError: cannot import name XPathParserBase
What exactly is the bug? Do you not expect that ImportError to be raised? Or is it just that the path names shown in tracebacks are wrong?
Further investigation turns up the culprit: The generated pyc files from the RPM have the "/var/tmp/..." hard-coded in to their imports.
Presumably bdist_rpm shouldn't contain .pyc files, and should instead use a postinstall script to compile the .py files.
--amk
Andrew Kuchling wrote:
line 3, in ? ImportError: cannot import name XPathParserBase
What exactly is the bug? Do you not expect that ImportError to be raised? Or is it just that the path names shown in tracebacks are wrong?
The ImportError is being raised _because_ of the wrong pathnames. If I perform the exact same procedure on the exact same machine with a 4Suite installed from the distutils sdist, I get no such ImportError.
Further investigation turns up the culprit: The generated pyc files from the RPM have the "/var/tmp/..." hard-coded in to their imports.
Presumably bdist_rpm shouldn't contain .pyc files, and should instead use a postinstall script to compile the .py files.
Incorrect. Here is the .spec that bdist_rpm generates:
%define name 4Suite %define version 0.9.0 %define release 1
Summary: 4Suite Description Name: %{name} Version: %{version} Release: %{release} Source0: %{name}-%{version}.tar.gz Copyright: UNKNOWN Group: Development/Libraries BuildRoot: %{_tmppath}/%{name}-buildroot Prefix: %{_prefix} Vendor: Fourthought, Inc. support@4suite.org Url: http://www.4suite.org
%description 4Suite is a collection of Python tools for XML processing and object-databases.
4Suite is an integrated packaging of 4DOM, 4XPath, 4XSLT, 4RDF, and 4ODS.
%prep %setup
%build env CFLAGS="$RPM_OPT_FLAGS" python setup.py build
%install python setup.py install --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES
%clean rm -rf $RPM_BUILD_ROOT
%files -f INSTALLED_FILES %defattr(-,root,root)
As you can see, there is no post-installation script registration.
Note that I can probably solve this probblem by building the RPM by hand without the BuildRoot line or the "--root=$RPM_BUILD_ROOT" argument in the "python setup.py...". Of course I'll need a smarter %clean section as well.
If so, I'll post a working .spec file for additional data.
In case it helps anyone to look at the package, there is a 4Suite sdist available at
ftp://ftp.fourthought.com/pub/4Suite/4Suite-0.9.0rc1.tar.gz
Note the prerequisite of
ftp://ftp.fourthought.com/pub/4Suite/BisonGen-0.5.0rc1.tar.gz
On 20 September 2000, Uche Ogbuji said:
First the symptom. I build 4Suite using distutils. The sdist, bdist and bdist_win packages work just fine, but when I move the generated RPM to another machine and try to use 4Suite:
[uogbuji@dynip-23 uogbuji]$ python Python 1.5.2 (#1, Jul 12 2000, 10:39:14) [GCC 2.96 20000702 (experimental)] on linux-i386 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
import sys sys.path
['', '/usr/lib/python1.5/', '/usr/lib/python1.5/plat-linux-i386', '/usr/lib/python1.5/lib-tk', '/usr/lib/python1.5/lib-dynload', '/usr/lib/python1.5/site-packages']
import xml.xslt
Traceback (innermost last): File "<stdin>", line 1, in ? File "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xslt/__init__.py", line 3, in ? File "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xpath/__init__.py", line 3, in ? ImportError: cannot import name XPathParserBase
The problem is the phantom "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages"...
That's bizarre. Having the wrong filename in the .pyc files is annoying, but I can't for the life of me see how it could be causing an ImportError.
I've downloaded your two source dists to see if I can figure out what the heck's going on. Don't have anything yet, but I'm glad to see someone is using all those extensibility hooks I put in! AFAIK you're the first ones to define a new Distribution class -- cool! Thought I do wonder if all the heavy lifting you're doing to beat the Distutils into submission could be causing problems. It's idle speculation right now; I haven't even reproduced your problems yet, much less tried to debug them. Tomorrow... my brain is shutting down for the night right now.
Greg
Greg Ward wrote:
"/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xpath/__init__.py",
line 3, in ? ImportError: cannot import name XPathParserBase
The problem is the phantom "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages"...
That's bizarre. Having the wrong filename in the .pyc files is annoying, but I can't for the life of me see how it could be causing an ImportError.
We just found out that this is indeed not the problem. The problem was that RPM wasn't getting the right lists of installed files, so it was getting confused about where things were.
I've downloaded your two source dists to see if I can figure out what the heck's going on. Don't have anything yet, but I'm glad to see someone is using all those extensibility hooks I put in! AFAIK you're the first ones to define a new Distribution class -- cool!
You ain't seen nothing yet! We fixed a whole bunch of our additional packaging problems with more distutils subclassing. Check out
ftp://ftp.fourthought.com/pub/4Suite/4Suite-0.9.0.2.tar.gz
For the sordid mess...
Thought I do wonder if all the heavy lifting you're doing to beat the Distutils into submission could be causing problems. It's idle speculation right now; I haven't even reproduced your problems yet, much less tried to debug them. Tomorrow... my brain is shutting down for the night right now.
We'd love to hear any feedback about our distutils efforts. We hardly thought we were doing anything extraordinary, but distutils does seem to have several limitations as is that we had to sub-class to get around. Maybe we were trying too hard in some cases.
Uche Ogbuji wrote:
Greg Ward wrote:
We'd love to hear any feedback about our distutils efforts. We hardly thought we were doing anything extraordinary, but distutils does seem to have several limitations as is that we had to sub-class to get around. Maybe we were trying too hard in some cases.
There were a couple of spots were if we could abstract core distutils code into a function call (that we could override) it would make our lives that much nicer. Ex, I needed on little hook into install_lib.run (after the build and before the copy), but I had to reproduce the entire function call...
another really useful thing for us on copy_file would be if the function told you if the file was copied....
Mike
-- Uche Ogbuji Principal Consultant uche.ogbuji@fourthought.com +1 303 583 9900 x 101 Fourthought, Inc. http://Fourthought.com 4735 East Walnut St, Ste. C, Boulder, CO 80301-2537, USA Software-engineering, knowledge-management, XML, CORBA, Linux, Python
Distutils-SIG maillist - Distutils-SIG@python.org http://www.python.org/mailman/listinfo/distutils-sig
On 21 September 2000, Mike Olson said:
There were a couple of spots were if we could abstract core distutils code into a function call (that we could override) it would make our lives that much nicer. Ex, I needed on little hook into install_lib.run (after the build and before the copy), but I had to reproduce the entire function call...
OK, I've split install_lib.run() up. It's now a three-liner:
self.build() outfiles = self.install() self.bytecompile(outfiles)
and I have no problems with making people cut-n-paste three lines of code in order to insert one. ;-) I *think* this approach is preferable to sticking in "pre_build()" and "post_install()" hooks, which are empty methods by default.
Try this out with the CVS version, but don't count on the interface staying the same -- ie. that 'bytecompile()' method might disappear, if byte-compilation moves to the build phase. Whatever goes out in 0.9.3 should be final.
another really useful thing for us on copy_file would be if the function told you if the file was copied....
Hmm, every time this comes up I can never remember why I didn't change it the last time around. Or maybe I did change it last time, and now you want the old behaviour back. Argh! Good thing I have CVS here to cover my you-know-what... ah! here we go:
revision 1.4 date: 2000/06/23 01:42:40; author: gward; state: Exp; lines: +6 -7 Bastian Kleineidam: 'copy_file()' now returns the output filename, rather than a boolean indicating whether it did the copy.
This bloody return value has bounced back and forth between "was it copied?" and "output filename", and I'm *not* changing it again! It makes my head hurt, which is not worth it over such a silly, trivial little thing. There is code out there that relies on getting the output filename back, regardless of whether the file was copied or not.
Greg
Greg Ward wrote:
On 21 September 2000, Mike Olson said:
There were a couple of spots were if we could abstract core distutils code into a function call (that we could override) it would make our lives that much nicer. Ex, I needed on little hook into install_lib.run (after the build and before the copy), but I had to reproduce the entire function call...
OK, I've split install_lib.run() up. It's now a three-liner:
self.build() outfiles = self.install() self.bytecompile(outfiles)and I have no problems with making people cut-n-paste three lines of code in order to insert one. ;-) I *think* this approach is preferable to sticking in "pre_build()" and "post_install()" hooks, which are empty methods by default.
Try this out with the CVS version, but don't count on the interface staying the same -- ie. that 'bytecompile()' method might disappear, if byte-compilation moves to the build phase. Whatever goes out in 0.9.3 should be final.
This looks like it will help, I'll try out the latest CVS version today yet.
another really useful thing for us on copy_file would be if the function told you if the file was copied....
Hmm, every time this comes up I can never remember why I didn't change it the last time around. Or maybe I did change it last time, and now you want the old behaviour back. Argh! Good thing I have CVS here to cover my you-know-what... ah! here we go:
revision 1.4 date: 2000/06/23 01:42:40; author: gward; state: Exp; lines: +6 -7 Bastian Kleineidam: 'copy_file()' now returns the output filename, rather than a boolean indicating whether it did the copy.
This bloody return value has bounced back and forth between "was it copied?" and "output filename", and I'm *not* changing it again! It makes my head hurt, which is not worth it over such a silly, trivial little thing. There is code out there that relies on getting the output filename back, regardless of whether the file was copied or not.
I've noticed the code that relies on copy_file returning the dest file. I also understand the need to have that code in place. However, (even though this could be a futile attmept), what about returning a tuple, best of both worlds, or even better, a new interface that returns the tuple. Rename copy_file to the new interface name and modify it to return the tuple of dest name and a boolean copied flag. Then recreate the function copy_file to work as it does.....
def new_copy_file(src,dest,....):
return (dest,copied)
def copy_file(src,dest,....): return new_copy_file(src,dest,...)[0]
We'd also want to do the same for copy_tree where is can return a list of tuples....
Just a thought...
Mike
Greg
On 21 September 2000, Uche Ogbuji said:
We just found out that this is indeed not the problem. The problem was that RPM wasn't getting the right lists of installed files, so it was getting confused about where things were.
...which probably has something to do with your highly customized build process...
You ain't seen nothing yet! We fixed a whole bunch of our additional packaging problems with more distutils subclassing. Check out
ftp://ftp.fourthought.com/pub/4Suite/4Suite-0.9.0.2.tar.gz
...OK, I'll download this one once I've got SWIG installed. *Then* I'll look into your sordid mess!
We'd love to hear any feedback about our distutils efforts. We hardly thought we were doing anything extraordinary, but distutils does seem to have several limitations as is that we had to sub-class to get around. Maybe we were trying too hard in some cases.
No build system can cover every possibility out of the box, that's why they have to be extensible. Distutils covers pure Python modules, vanilla Python extensions, SWIGged Python extensions, and C libraries that are statically linked into Python extensions pretty nicely (if I do say so myself). Anything outside of that box and you've got to write custom code.
(The theory was that Python programmers would prefer to write such custom code in Python, rather than cobbling together a nightmarish mix of m4, shell, and make that a traditional Autoconf/Automake system implies -- and of course, doing it all in Python makes it portable to other platforms. In practice, it turns out that extending a large object framework requires a bit more in-depth knowledge than cranking out a few more Makefile targets. Well, if nothing else it's an interesting experiment, and I think it's fairly successful.)
Anyways, I already got some feedback from Mike about refactoring the install_lib command for easier extensibility -- and in fact I'm about to checkin a change that should help there. See my next post.
Greg
participants (4)
-
Andrew Kuchling -
Greg Ward -
Mike Olson -
Uche Ogbuji