[Distutils] First steps with distutils...
M.-A. Lemburg
mal@lemburg.com
Mon Sep 4 17:51:19 2000
Greg Ward wrote:
>
> On 04 September 2000, M.-A. Lemburg said:
> > > Correct. MANIFEST is a simple list of files, one per line. If you want
> > > to include whole directories, you should write a MANIFEST.in file.
> >
> > I didn't mean that distutils should scan the goven directory --
> > it should simply ignore it instead of raising an exception.
>
> Hmm, are you really getting an exception? When I put a directory name
> into a MANIFEST file, I get:
>
> error: can't copy 'pkg': doesn't exist or not a regular file
>
> and immediate termination of the setup script (hey, it's an error, not a
> warning). I'll admit that's not as useful as it could be, but as long
> as there's no traceback it's not a bug! ;-)
You're right; it's not an exception, but still something which
causes distutils to stop ;-)
Would it hurt much adding a "continue" somewhere to remedy
this minor "caveat" :-)
> Actually, I spent some time documenting the "sdist" command today, and
> tracked down some bugs in the handling of the manifest files. I also
> now understand the rules a bit better -- amazing what reading the code
> will tell you. ;-) Here are the rules as implemented in the current
> code (not even checked in yet, because of the 2.0b1 code freeze):
>
> cases:
> 1) no manifest, template exists: generate manifest
> (covered by 2a: no manifest == template newer)
> 2) manifest & template exist:
> 2a) template or setup script newer than manifest:
> regenerate manifest
> 2b) manifest newer than both:
> do nothing (unless --force or --manifest-only)
> 3) manifest exists, no template:
> do nothing (unless --force or --manifest-only)
> 4) no manifest, no template: generate w/ warning ("defaults only")
>
> In the 0.9.2 code (and all Distutils releases since at least 0.2), case
> (4) wasn't handled properly. A workaround is to use the -f
> (--force-manifest) option, which causes the sdist command to
> unconditionally regenerate the MANIFEST file.
>
> Also, case (3) implies that the "default file set" -- setup.py,
> README.txt, and any source code specified in the setup script -- will
> *not* be used if you supply your own MANIFEST file. Additionally, the
> default "prune list" -- which strips RCS and CVS directories, as well as
> Distutils-generated temporary directories -- does not apply. IOW, if
> you generate your own MANIFEST, you have to get it exactly right. I
> think this is the right thing to do, because only control freaks will
> want to supply their own MANIFEST file, and taking away any control from
> control freaks makes them, well, freak out. ;-)
True ;-)
> See the 'get_file_list()' method in distutils/command/sdist.py if you're
> curious about the implementation. (And see if you can spot the bug that
> I fixed this morning.) (No fair watching python-checkins!)
>
> > I'm currently using my own tools for generating the MANIFEST
> > file. The most important difference is that they allow per
> > directory MANIFEST.in style files which are appended to the
> > general MANIFEST.in logic while scanning the directory.
>
> Hmmm, another good reason to roll your own manifest -- maybe it's not
> just for control freaks?
>
> > My version if rpm 3.0.3.
> [...]
> > I also noted another bug:
> > When building an RPM which contains more than one Extension(),
> > bdist_rpm fails on the second Extension(): it can't find
> > the C file.
> > """
> > running build_ext
> > building 'mx.DateTime.mxDateTime.mxDateTime' extension
> > creating build/temp.linux2
> > creating build/temp.linux2/mx
> > creating build/temp.linux2/mx/DateTime
> > creating build/temp.linux2/mx/DateTime/mxDateTime
> > gcc -g -O2 -fpic -Imx/DateTime/mxDateTime -I/usr/local/include/python2.0 -c mx/DateTime/mxDateTime/mxDateTime.c -o build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -O2 -m486 -fno-strength-reduce
> > creating build/lib.linux2/mx/DateTime
> > creating build/lib.linux2/mx/DateTime/mxDateTime
> > gcc -shared build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -o build/lib.linux2/mx/DateTime/mxDateTime/mxDateTime.so
> > building 'mx.Proxy.mxProxy.mxProxy' extension
> > error: file 'mx/Proxy/mxProxy/mxProxy.c' does not exist
> > Bad exit status from /var/tmp/rpm-tmp.85854 (%build)
> > error: command 'rpm' failed with exit status 1
> > """
>
> ??? Very weird. Does "python setup.py build" work on its own, ie. when
> run by you rather than by rpm in its build directory?
Yes. Works just fine.
> One simple possibility is that you forgot to include
> mx/Proxy/mxProxy/mxProxy.c in your tarball -- make sure that you can
> unpack your tarball in a fresh directory and run "python setup.py
> build", which is just what rpm is doing.
That could have been the cause. While testing this I stumbled
over another bugglet:
Executing: %build
+ umask 022
+ cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD
+ cd mx-Extensions-BASE-1.0.0
+ env 'CFLAGS=-O2 -m486 -fno-strength-reduce' python setup.py build
Traceback (innermost last):
File "setup.py", line 7, in ?
from distutils.core import setup, Extension
File "/home/lemburg/lib/distutils/core.py", line 146
raise
^
SyntaxError: invalid syntax
This is due to RPM finding a Python 1.5 version under the
simple name "python".... distutils doesn't seem to be compatible
with multiple installed Python versions :-( [I did run the
setup.py file using Python 2.0].
I then tried:
projects/tmp> python2.0 setup.py bdist_rpm --prep-script ./prep-script
invalid command name './prep-script'
Looks like somethings wrong there: I can't seem to pass a script
filename to the command.
I had to hack the bdist_rpm.py file to make it work (perhaps time
for yet another option ?-)
...
OK, after that hack the RPM compile runs through, but it now finishes
with the following lines (and no apparent reason):
creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools
creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools
copying build/lib.linux2/mx/Tools/mxTools/mxTools.so -> /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools
byte-compiling /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/__init__.py to __init__.pyc
writing list of installed files to 'INSTALLED_FILES'
warning: install: modules installed to '/var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself
+ exit 0
Processing files: mx-Extensions-BASE
File listed twice: /usr/local/lib/python2.0/site-packages/mx/Tools/mxTools/mxTools.so
Finding provides...
Finding requires...
Provides: mxDateTime.so mxProxy.so mxQueue.so mxStack.so mxTextTools.so mxTools.so
Requires: ld-linux.so.2 libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1)
error: command 'rpm' failed with exit status 1
Does this mean that my system doesn't provide the "required"
packages ? (SuSE has a somewhat different naming scheme than
RedHat so this could be a possible cause)
> > > For a straight Distutil build (ie. no RPM involved), setting CFLAGS
> > > should work, but it only applies to extensions (not C libraries). When
> > > RPM is involved, I'm not sure what the Right Way to do it is. Maybe
> > > CFLAGS will work? Maybe editing /etc/rpmrc or /usr/lib/rpm/rpmrc? I
> > > dunno.
> >
> > Wouldn't an option to distutils be the right way for these kind
> > of things ?
>
> Yes. Definitely. It's on the TODO list.
Great :-)
> Now, on to your other "bdist_rpm" failure...
>
> > setup.py bdist_rpm output:
> >
> > rpm -ba --define _topdir /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm --clean build/bdist.linux2/rpm/SPECS/mx-Extensions-BASE.spec
> > Executing: %prep
> [...]
> > Wrote: /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/SRPMS/mx-Extensions-BASE-1.0.0-1.src.rpm
> > Could not open /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386/mx-Extensions-BASE-1.0.0-1.i386.rpm
> >
> > error: command 'rpm' failed with exit status 1
>
> Hmmm, this smells like the archetypal Distutils bug: someone's trying to
> write a file into a directory that doesn't exist yet. In this case, it
> would be rpm's fault, since the "bdist_rpm" command doesn't know the
> name of the architecture directory that rpm will try to write to.
> Things to look for:
> * does /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386
> exist?
No.
> * if you create it, does the bdist_rpm command then run successfully?
Yes.
> I haven't had problems with rpm failing to create directories -- I have
> RPM 3.0.4 on my Red Hat 6.2 box, and I just cranked out an RPM of
> mxDateTime 1.3.0 without a hitch. Perhaps RPM 3.0.3 joins RPM 2.x on
> the index of forbidden versions... sigh...
Dunno. It did work after I manually created the i386 directory
and applied the python2.0 hack to bdist_rpm.py.
Thanks,
--
Marc-Andre Lemburg
______________________________________________________________________
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/