Mailman 3 First steps with distutils... - Distutils-SIG

First steps with distutils...

M.-A. Lemburg

1 Sep 2000 1 Sep '00

10:51 p.m.

Here is a list of things I've found while playing with the 0.9.2 snapshot. I've also posted this to Greg directly, but on second thought I think it might be worthwhile getting some feedback from distutilers as well: * distutils doesn't seem to like diretory names in MANIFEST * the setup.py file itself is not included in the source distribution per default (why not ?) * bdist_wininst wants zlib but the installer has a note that it is not using it anymore * bdist_wininst fails with a setup.py file which does not define long_description * bdist_rpm fails with an error in the rpm command (the source rpm is built, but it fails to build to the binary rpm for some reason -- it says, that it can't find the file ??); do I need some new rpm release for this to work ? * the bdist commands don't compile the .py files into .pyo but this is needed in order to use them with python -O * bdist (the dumb version) defaults to creating a tar.gz archive which is relative to / -- it shouldn't use the usr/local/ prefix to make the installation relocateable (e.g. to install under /usr, /opt, /opt/local, etc.) * how can I define the compiler settings for extension compilation (it defaults to -m486 for RPMs -- -mcpu=pentium would be more appropriate; default for standard bdist seems to be -O2 and no architecture) ? And a general question: how can I define which files are documentation, examples and auxiliary files ? More to come :-) -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Show replies by thread

Greg Ward

2 Sep 2 Sep

3:52 a.m.

On 02 September 2000, M.-A. Lemburg said:

...

I've also posted this to Greg directly, but on second thought I think it might be worthwhile getting some feedback from distutilers as well:

Then I'll reply publicly too; seems more useful that way.

...

* distutils doesn't seem to like diretory names in MANIFEST

Correct. MANIFEST is a simple list of files, one per line. If you want to include whole directories, you should write a MANIFEST.in file. Eg. here is the Distutils' MANIFEST.in; I have (right now) 536 files in my Distutils working tree, and this MANIFEST.in whittles it down to the 89 files you see in the 0.9.2 source tarball: include *.txt TODO include MANIFEST.in recursive-include examples *.txt *.py prune examples/sample*/build include doc/dist/*.tex doc/inst/*.tex graft misc exclude misc/*.zip global-exclude *~ The docs for MANIFEST.in are sketchy right now; see http://www.python.org/sigs/distutils-sig/doc/dist/sdist-cmd.html for a start.

...

* the setup.py file itself is not included in the source distribution per default (why not ?)

It should be if you have a MANIFEST.in file. One current weakness is that the "sdist" command isn't very smart in the absence of a MANIFEST.in file. I always seem to notice this in the final hours up to a release, which is not the time or place for major surgery to fix this. Sigh.

...

* bdist_wininst wants zlib but the installer has a note that it is not using it anymore

Thomas, is this true?

...

* bdist_wininst fails with a setup.py file which does not define long_description

I thought that was fixed in 0.9.2 -- !@&%@!#&!@#&%!

...

* bdist_rpm fails with an error in the rpm command (the source rpm is built, but it fails to build to the binary rpm for some reason -- it says, that it can't find the file ??); do I need some new rpm release for this to work ?

bdist_rpm requires RPM 3.0, which all current RPM-based Linux distributions use (AFAIK). It does *not* work on Red Hat 5.x, which is what finally prompted me to upgrade to 6.2. ;-) Can you show the exact output when you run "setup.py bdist_rpm"?

...

* the bdist commands don't compile the .py files into .pyo but this is needed in order to use them with python -O

Yeah, known problem. How big do you want your RPM to be again? ;-(

...

* bdist (the dumb version) defaults to creating a tar.gz archive which is relative to / -- it shouldn't use the usr/local/ prefix to make the installation relocateable (e.g. to install under /usr, /opt, /opt/local, etc.)

There's no one right way to do this; I've tried them both, and they're both wrong. Originally, bdist_dumb created an archive relative to <prefix> -- relocatable, but it didn't work if prefix != exec-prefix. (I can't remember if it bombed or created a bogus archive, but the bottom line is it didn't work.) Now it creates an archive relative to /, which handles prefix != exec-prefix, but is not relocatable. ;-( I think the answer is to let the packager decide, ie. add an option to make the archive relative to / or prefix. (And it should blow up if prefix != exec-prefix and you try to make a prefix-relative archive.)

...

* how can I define the compiler settings for extension compilation (it defaults to -m486 for RPMs -- -mcpu=pentium would be more appropriate; default for standard bdist seems to be -O2 and no architecture) ?

For a straight Distutil build (ie. no RPM involved), setting CFLAGS should work, but it only applies to extensions (not C libraries). When RPM is involved, I'm not sure what the Right Way to do it is. Maybe CFLAGS will work? Maybe editing /etc/rpmrc or /usr/lib/rpm/rpmrc? I dunno.

...

And a general question: how can I define which files are documentation, examples and auxiliary files ?

Doc files can be defined with the doc_files option to bdist_rpm. The easiest way to set this is with a setup.cfg in your distribution root, eg. the Distutils setup.cfg has: [bdist_rpm] release = 1 packager = Greg Ward doc_files = CHANGES.txt README.txt USAGE.txt doc/ examples/ ... Although I have been thinking that putting making every RPM option a Distutils option was a bad idea. Hmmmm.

...

More to come :-)

Oh no! You might want to take a look in the Distutils-0.9.2/TODO file to see what's on "the list". Greg -- Greg Ward - nerd gward@python.net http://starship.python.net/~gward/ Does your DRESSING ROOM have enough ASPARAGUS?

Harry Henry Gebel

4:46 a.m.

On Sat, Sep 02, 2000 at 11:49:56AM +0200, M.-A. Lemburg wrote:

...

* bdist_rpm fails with an error in the rpm command (the source rpm is built, but it fails to build to the binary rpm for some reason -- it says, that it can't find the file ??); do I need some new rpm release for this to work ?

bdist_rpm requires RPM version 3, there was some discussion about supporting RPM 2; but it did not seem practical to me to support RPM 2 in a way that would work across different RPM based platforms. It probably would not be hard to make it produce version 2 spec files, but the packager would then have to build them outside of the Distutils.

...

* the bdist commands don't compile the .py files into .pyo but this is needed in order to use them with python -O

There were some attempts made to do this, but none has been satisfactory.

...

* how can I define the compiler settings for extension compilation (it defaults to -m486 for RPMs -- -mcpu=pentium would be more appropriate; default for standard bdist seems to be -O2 and no architecture) ?

The compiler options for RPMs are set by RPM (using CFLAGS) not by Distutils, you can change these by modifying your RPM configuration (the exact modifications depend on which version of RPM you are using.) Otherwise you can use CFLAGS to set them, I think they can be set in setup.py, but I'm not sure about that. -- Harry Henry Gebel, Senior Developer, Landon House SBS ICQ# 76308382 West Dover Hundred, Delaware

M.-A. Lemburg

4 Sep 4 Sep

12:13 a.m.

Greg Ward wrote:

...

On 02 September 2000, M.-A. Lemburg said:

...
* distutils doesn't seem to like diretory names in MANIFEST

Correct. MANIFEST is a simple list of files, one per line. If you want to include whole directories, you should write a MANIFEST.in file.

I didn't mean that distutils should scan the goven directory -- it should simply ignore it instead of raising an exception. The reason is that generation tools for MANIFEST could sometimes include directory entries in the list (e.g. take a ZIP file directory listing or ls -1p).

...

Eg. here is the Distutils' MANIFEST.in; I have (right now) 536 files in my Distutils working tree, and this MANIFEST.in whittles it down to the 89 files you see in the 0.9.2 source tarball:

include *.txt TODO include MANIFEST.in recursive-include examples *.txt *.py prune examples/sample*/build include doc/dist/*.tex doc/inst/*.tex graft misc exclude misc/*.zip global-exclude *~

The docs for MANIFEST.in are sketchy right now; see

http://www.python.org/sigs/distutils-sig/doc/dist/sdist-cmd.html

for a start.

I'm currently using my own tools for generating the MANIFEST file. The most important difference is that they allow per directory MANIFEST.in style files which are appended to the general MANIFEST.in logic while scanning the directory.

...

...
* the setup.py file itself is not included in the source distribution per default (why not ?)

It should be if you have a MANIFEST.in file.

Ahh, ok. So that's why it doesn't get included: I don't have a MANIFEST.in file.

...

One current weakness is that the "sdist" command isn't very smart in the absence of a MANIFEST.in file. I always seem to notice this in the final hours up to a release, which is not the time or place for major surgery to fix this. Sigh.

...
* bdist_wininst wants zlib but the installer has a note that it is not using it anymore

Thomas, is this true?

I would make the usage of zlib optional with distutils defaulting to not using it (since it isn't enabled by default in the Python distribution).

...

...
* bdist_wininst fails with a setup.py file which does not define long_description

I thought that was fixed in 0.9.2 -- !@&%@!#&!@#&%!

I used the latest snapshot, not the final 0.9.2 release version. I'll check this again.

...

...
* bdist_rpm fails with an error in the rpm command (the source rpm is built, but it fails to build to the binary rpm for some reason -- it says, that it can't find the file ??); do I need some new rpm release for this to work ?

bdist_rpm requires RPM 3.0, which all current RPM-based Linux distributions use (AFAIK). It does *not* work on Red Hat 5.x, which is what finally prompted me to upgrade to 6.2. ;-)

My version if rpm 3.0.3.

...

Can you show the exact output when you run "setup.py bdist_rpm"?

I'll attach a copy below. I also noted another bug: When building an RPM which contains more than one Extension(), bdist_rpm fails on the second Extension(): it can't find the C file. """ running build_ext building 'mx.DateTime.mxDateTime.mxDateTime' extension creating build/temp.linux2 creating build/temp.linux2/mx creating build/temp.linux2/mx/DateTime creating build/temp.linux2/mx/DateTime/mxDateTime gcc -g -O2 -fpic -Imx/DateTime/mxDateTime -I/usr/local/include/python2.0 -c mx/DateTime/mxDateTime/mxDateTime.c -o build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -O2 -m486 -fno-strength-reduce creating build/lib.linux2/mx/DateTime creating build/lib.linux2/mx/DateTime/mxDateTime gcc -shared build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -o build/lib.linux2/mx/DateTime/mxDateTime/mxDateTime.so building 'mx.Proxy.mxProxy.mxProxy' extension error: file 'mx/Proxy/mxProxy/mxProxy.c' does not exist Bad exit status from /var/tmp/rpm-tmp.85854 (%build) error: command 'rpm' failed with exit status 1 """

...

...
* the bdist commands don't compile the .py files into .pyo but this is needed in order to use them with python -O

Yeah, known problem. How big do you want your RPM to be again? ;-(

I don't really care for the size of the file as long as it runs in all possible settings (don't want a FAQ along the lines of "mx.DateTime fails to import..."). A simple solution would be to compile the Python code using os.system('python -O compileall.py') with the optimization level being a distutils option.

...

...
* bdist (the dumb version) defaults to creating a tar.gz archive which is relative to / -- it shouldn't use the usr/local/ prefix to make the installation relocateable (e.g. to install under /usr, /opt, /opt/local, etc.)

There's no one right way to do this; I've tried them both, and they're both wrong. Originally, bdist_dumb created an archive relative to <prefix> -- relocatable, but it didn't work if prefix != exec-prefix. (I can't remember if it bombed or created a bogus archive, but the bottom line is it didn't work.) Now it creates an archive relative to /, which handles prefix != exec-prefix, but is not relocatable. ;-(

I think the answer is to let the packager decide, ie. add an option to make the archive relative to / or prefix. (And it should blow up if prefix != exec-prefix and you try to make a prefix-relative archive.)

That would be ok.

...

...
* how can I define the compiler settings for extension compilation (it defaults to -m486 for RPMs -- -mcpu=pentium would be more appropriate; default for standard bdist seems to be -O2 and no architecture) ?

For a straight Distutil build (ie. no RPM involved), setting CFLAGS should work, but it only applies to extensions (not C libraries). When RPM is involved, I'm not sure what the Right Way to do it is. Maybe CFLAGS will work? Maybe editing /etc/rpmrc or /usr/lib/rpm/rpmrc? I dunno.

Wouldn't an option to distutils be the right way for these kind of things ?

...

...
And a general question: how can I define which files are documentation, examples and auxiliary files ?

Doc files can be defined with the doc_files option to bdist_rpm. The easiest way to set this is with a setup.cfg in your distribution root, eg. the Distutils setup.cfg has:

[bdist_rpm] release = 1 packager = Greg Ward doc_files = CHANGES.txt README.txt USAGE.txt doc/ examples/ ...

Although I have been thinking that putting making every RPM option a Distutils option was a bad idea. Hmmmm.

Well, documentation is usually installed under /usr/doc/packages so this option is important. Perhaps you could add a generic pass-through-to-rpm option (much like the -Wl,<options> option on gcc) ?!

...

...
More to come :-)

Oh no! You might want to take a look in the Distutils-0.9.2/TODO file to see what's on "the list".

Will do. Thanks for the feedback.

...

Greg -- Greg Ward - nerd gward@python.net http://starship.python.net/~gward/ Does your DRESSING ROOM have enough ASPARAGUS?

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://www.python.org/mailman/listinfo/distutils-sig

-- setup.py bdist_rpm output: rpm -ba --define _topdir /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm --clean build/bdist.linux2/rpm/SPECS/mx-Extensions-BASE.spec Executing: %prep + umask 022 + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD + rm -rf mx-Extensions-BASE-1.0.0 + tar -xvvf - + /bin/gzip -dc /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/SOURCES/mx-Extensions-BASE-1.0.0.tar.gz drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/ drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/mx/ drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/mx/DateTime/ drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/mx/DateTime/Doc/ -rw-r--r-- lemburg/users 12532 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Doc/mxDateTime-History.html -rw-r--r-- lemburg/users 103704 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Doc/mxDateTime.html drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/ -rw-r--r-- lemburg/users 1577 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/AtomicClock.py -rw-r--r-- lemburg/users 12973 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/CommandLine.py -rw-r--r-- lemburg/users 399 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/Y2000.py -rw-r--r-- lemburg/users 0 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/__init__.py -rwxr-xr-x lemburg/users 1465 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/alarm.py -rw-r--r-- lemburg/users 564 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Examples/lifespan.py drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/ -rw-r--r-- lemburg/users 10228 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/Makefile.pre.in -rw-r--r-- lemburg/users 655 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/Setup.in -rw-r--r-- lemburg/users 155 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/W95.bat -rw-r--r-- lemburg/users 521 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/__init__.py -rwxr-xr-x lemburg/users 5965 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/ld_so_aix -rwxr-xr-x lemburg/users 3101 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/makexp_aix -rw-r--r-- lemburg/users 19753 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mx.h -rw-r--r-- lemburg/users 99513 2000-09-01 15:08 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime.c -rw-r--r-- lemburg/users 24 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime.def -rw-r--r-- lemburg/users 10058 2000-09-01 15:08 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime.h -rw-r--r-- lemburg/users 7039 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime.mak -rwxr-xr-x lemburg/users 57344 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime.pyd -rw-r--r-- lemburg/users 24804 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime.sit.hqx -rw-r--r-- lemburg/users 18248 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxDateTime_Python.py -rw-r--r-- lemburg/users 1554 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxh.h -rw-r--r-- lemburg/users 8552 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxpyapi.h -rw-r--r-- lemburg/users 6903 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/mxstdlib.h -rw-r--r-- lemburg/users 9202 2000-09-01 15:08 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/test.py -rw-r--r-- lemburg/users 426 2000-09-01 15:08 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/testcmp.py -rw-r--r-- lemburg/users 385 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/testcomdates.py -rw-r--r-- lemburg/users 147 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/teststrftime.py -rw-r--r-- lemburg/users 1358 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/testticks.py -rw-r--r-- lemburg/users 480 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/mxDateTime/win32mk.bat -rw-r--r-- lemburg/users 7290 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/ARPA.py -rw-r--r-- lemburg/users 26902 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/DateTime.py -rw-r--r-- lemburg/users 2724 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Feasts.py -rw-r--r-- lemburg/users 10374 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/ISO.py -rw-r--r-- lemburg/users 143 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/LICENSE -rw-r--r-- lemburg/users 2561 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/LazyModule.py -rw-r--r-- lemburg/users 3760 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Locale.py -rw-r--r-- lemburg/users 863 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Makefile.pkg -rw-r--r-- lemburg/users 14232 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/NIST.py -rw-r--r-- lemburg/users 5262 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/ODMG.py -rw-r--r-- lemburg/users 15580 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Parser.py -rw-r--r-- lemburg/users 142 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/README -rw-r--r-- lemburg/users 4353 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/Timezone.py -rw-r--r-- lemburg/users 2110 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/__init__.py -rw-r--r-- lemburg/users 2753 2000-09-01 15:09 mx-Extensions-BASE-1.0.0/mx/DateTime/timegm.py drwxr-xr-x lemburg/users 0 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/mx/Misc/ -rw-r--r-- lemburg/users 2561 2000-09-01 15:10 mx-Extensions-BASE-1.0.0/mx/Misc/LazyModule.py -rw-r--r-- lemburg/users 0 2000-09-01 15:10 mx-Extensions-BASE-1.0.0/mx/Misc/__init__.py -rw-r--r-- lemburg/users 891 2000-09-01 15:08 mx-Extensions-BASE-1.0.0/mx/__init__.py -rwxr-xr-x lemburg/users 1587 2000-09-04 13:01 mx-Extensions-BASE-1.0.0/setup.py + STATUS=0 + '[' 0 -ne 0 ']' + cd mx-Extensions-BASE-1.0.0 ++ /usr/bin/id -u + '[' 500 = 0 ']' ++ /usr/bin/id -u + '[' 500 = 0 ']' + /bin/chmod -Rf a+rX,g-w,o-w . + exit 0 Executing: %build + umask 022 + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD + cd mx-Extensions-BASE-1.0.0 + env 'CFLAGS=-O2 -m486 -fno-strength-reduce' python setup.py build running build running build_py creating build creating build/lib.linux2 creating build/lib.linux2/mx copying mx/__init__.py -> build/lib.linux2/mx running build_ext building 'mx.DateTime.mxDateTime.mxDateTime' extension creating build/temp.linux2 creating build/temp.linux2/mx creating build/temp.linux2/mx/DateTime creating build/temp.linux2/mx/DateTime/mxDateTime gcc -g -O2 -fpic -Imx/DateTime/mxDateTime -I/usr/local/include/python2.0 -c mx/DateTime/mxDateTime/mxDateTime.c -o build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -O2 -m486 -fno-strength-reduce creating build/lib.linux2/mx/DateTime creating build/lib.linux2/mx/DateTime/mxDateTime gcc -shared build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -o build/lib.linux2/mx/DateTime/mxDateTime/mxDateTime.so + exit 0 Executing: %install + umask 022 + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD + cd mx-Extensions-BASE-1.0.0 + python setup.py install --root=/var/tmp/mx-Extensions-BASE-buildroot --record=INSTALLED_FILES running install running build running build_py not copying mx/__init__.py (output up-to-date) running build_ext skipping 'mx.DateTime.mxDateTime.mxDateTime' extension (up-to-date) running install_lib not copying build/lib.linux2/mx/__init__.py (output up-to-date) copying build/lib.linux2/mx/DateTime/mxDateTime/mxDateTime.so -> /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/DateTime/mxDateTime skipping byte-compilation of /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/__init__.py writing list of installed files to 'INSTALLED_FILES' warning: install: modules installed to '/var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself + exit 0 Processing files: mx-Extensions-BASE Finding provides... Finding requires... Provides: mxDateTime.so Requires: ld-linux.so.2 libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) Wrote: /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/SRPMS/mx-Extensions-BASE-1.0.0-1.src.rpm Could not open /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386/mx-Extensions-BASE-1.0.0-1.i386.rpm error: command 'rpm' failed with exit status 1 -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

M.-A. Lemburg

12:19 a.m.

Harry Henry Gebel wrote:

...

On Sat, Sep 02, 2000 at 11:49:56AM +0200, M.-A. Lemburg wrote:

...
* bdist_rpm fails with an error in the rpm command (the source rpm is built, but it fails to build to the binary rpm for some reason -- it says, that it can't find the file ??); do I need some new rpm release for this to work ?

bdist_rpm requires RPM version 3, there was some discussion about supporting RPM 2; but it did not seem practical to me to support RPM 2 in a way that would work across different RPM based platforms. It probably would not be hard to make it produce version 2 spec files, but the packager would then have to build them outside of the Distutils.

I'm using rpm 3.0.3. FYI, I've attached a copy of the shell output to my reply to Greg.

...

...
* the bdist commands don't compile the .py files into .pyo but this is needed in order to use them with python -O

There were some attempts made to do this, but none has been satisfactory.

See my reply to Greg... this should be possible via os.system('python -O').

...

...
* how can I define the compiler settings for extension compilation (it defaults to -m486 for RPMs -- -mcpu=pentium would be more appropriate; default for standard bdist seems to be -O2 and no architecture) ?

The compiler options for RPMs are set by RPM (using CFLAGS) not by Distutils, you can change these by modifying your RPM configuration (the exact modifications depend on which version of RPM you are using.) Otherwise you can use CFLAGS to set them, I think they can be set in setup.py, but I'm not sure about that.

Hmm, I would rather not have to edit system files for RPM since this would make the setup.py file rely on the system's config files. A distutils option would be my preferred solution. Should be simple to add: the option would just have to set os.environ['CFLAGS'] to whatever value is specified in setup.py. setup.py can then include all the necessary logic to determine the compiler flags (e.g. by checking the default setting of the compiler classes). Thanks, -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Thomas Heller

4:31 a.m.

...

...
...
* bdist_wininst wants zlib but the installer has a note that it is not using it anymore

Thomas, is this true?

I would make the usage of zlib optional with distutils defaulting to not using it (since it isn't enabled by default in the Python distribution).

...
...
* bdist_wininst fails with a setup.py file which does not define long_description

I thought that was fixed in 0.9.2 -- !@&%@!#&!@#&%!

The current CVS version, which is the same as 0.9.2, does NOT use zlib any longer, also long_description is no longer needed. Thomas

Greg Ward

8:53 a.m.

On 04 September 2000, M.-A. Lemburg said:

...

...
Correct. MANIFEST is a simple list of files, one per line. If you want to include whole directories, you should write a MANIFEST.in file.

I didn't mean that distutils should scan the goven directory -- it should simply ignore it instead of raising an exception.

Hmm, are you really getting an exception? When I put a directory name into a MANIFEST file, I get: error: can't copy 'pkg': doesn't exist or not a regular file and immediate termination of the setup script (hey, it's an error, not a warning). I'll admit that's not as useful as it could be, but as long as there's no traceback it's not a bug! ;-) Actually, I spent some time documenting the "sdist" command today, and tracked down some bugs in the handling of the manifest files. I also now understand the rules a bit better -- amazing what reading the code will tell you. ;-) Here are the rules as implemented in the current code (not even checked in yet, because of the 2.0b1 code freeze): cases: 1) no manifest, template exists: generate manifest (covered by 2a: no manifest == template newer) 2) manifest & template exist: 2a) template or setup script newer than manifest: regenerate manifest 2b) manifest newer than both: do nothing (unless --force or --manifest-only) 3) manifest exists, no template: do nothing (unless --force or --manifest-only) 4) no manifest, no template: generate w/ warning ("defaults only") In the 0.9.2 code (and all Distutils releases since at least 0.2), case (4) wasn't handled properly. A workaround is to use the -f (--force-manifest) option, which causes the sdist command to unconditionally regenerate the MANIFEST file. Also, case (3) implies that the "default file set" -- setup.py, README.txt, and any source code specified in the setup script -- will *not* be used if you supply your own MANIFEST file. Additionally, the default "prune list" -- which strips RCS and CVS directories, as well as Distutils-generated temporary directories -- does not apply. IOW, if you generate your own MANIFEST, you have to get it exactly right. I think this is the right thing to do, because only control freaks will want to supply their own MANIFEST file, and taking away any control from control freaks makes them, well, freak out. ;-) See the 'get_file_list()' method in distutils/command/sdist.py if you're curious about the implementation. (And see if you can spot the bug that I fixed this morning.) (No fair watching python-checkins!)

...

I'm currently using my own tools for generating the MANIFEST file. The most important difference is that they allow per directory MANIFEST.in style files which are appended to the general MANIFEST.in logic while scanning the directory.

Hmmm, another good reason to roll your own manifest -- maybe it's not just for control freaks?

...

My version if rpm 3.0.3. [...] I also noted another bug: When building an RPM which contains more than one Extension(), bdist_rpm fails on the second Extension(): it can't find the C file. """ running build_ext building 'mx.DateTime.mxDateTime.mxDateTime' extension creating build/temp.linux2 creating build/temp.linux2/mx creating build/temp.linux2/mx/DateTime creating build/temp.linux2/mx/DateTime/mxDateTime gcc -g -O2 -fpic -Imx/DateTime/mxDateTime -I/usr/local/include/python2.0 -c mx/DateTime/mxDateTime/mxDateTime.c -o build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -O2 -m486 -fno-strength-reduce creating build/lib.linux2/mx/DateTime creating build/lib.linux2/mx/DateTime/mxDateTime gcc -shared build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -o build/lib.linux2/mx/DateTime/mxDateTime/mxDateTime.so building 'mx.Proxy.mxProxy.mxProxy' extension error: file 'mx/Proxy/mxProxy/mxProxy.c' does not exist Bad exit status from /var/tmp/rpm-tmp.85854 (%build) error: command 'rpm' failed with exit status 1 """

??? Very weird. Does "python setup.py build" work on its own, ie. when run by you rather than by rpm in its build directory? One simple possibility is that you forgot to include mx/Proxy/mxProxy/mxProxy.c in your tarball -- make sure that you can unpack your tarball in a fresh directory and run "python setup.py build", which is just what rpm is doing.

...

...
For a straight Distutil build (ie. no RPM involved), setting CFLAGS should work, but it only applies to extensions (not C libraries). When RPM is involved, I'm not sure what the Right Way to do it is. Maybe CFLAGS will work? Maybe editing /etc/rpmrc or /usr/lib/rpm/rpmrc? I dunno.

Wouldn't an option to distutils be the right way for these kind of things ?

Yes. Definitely. It's on the TODO list. Now, on to your other "bdist_rpm" failure...

...

setup.py bdist_rpm output:

rpm -ba --define _topdir /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm --clean build/bdist.linux2/rpm/SPECS/mx-Extensions-BASE.spec Executing: %prep [...] Wrote: /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/SRPMS/mx-Extensions-BASE-1.0.0-1.src.rpm Could not open /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386/mx-Extensions-BASE-1.0.0-1.i386.rpm

error: command 'rpm' failed with exit status 1

Hmmm, this smells like the archetypal Distutils bug: someone's trying to write a file into a directory that doesn't exist yet. In this case, it would be rpm's fault, since the "bdist_rpm" command doesn't know the name of the architecture directory that rpm will try to write to. Things to look for: * does /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386 exist? * if you create it, does the bdist_rpm command then run successfully? I haven't had problems with rpm failing to create directories -- I have RPM 3.0.4 on my Red Hat 6.2 box, and I just cranked out an RPM of mxDateTime 1.3.0 without a hitch. Perhaps RPM 3.0.3 joins RPM 2.x on the index of forbidden versions... sigh... Greg -- Greg Ward - Unix bigot gward@python.net http://starship.python.net/~gward/ If at first you don't succeed, redefine success.

Andrew Kuchling

9:47 a.m.

On Mon, Sep 04, 2000 at 03:50:56PM -0400, Greg Ward wrote:

...

I haven't had problems with rpm failing to create directories -- I have RPM 3.0.4 on my Red Hat 6.2 box, and I just cranked out an RPM of mxDateTime 1.3.0 without a hitch. Perhaps RPM 3.0.3 joins RPM 2.x on the index of forbidden versions... sigh...

I think it does; I noticed this problem on my machine, kronos, which has RPM 3.0.3 installed. --amk

M.-A. Lemburg

10:51 a.m.

Greg Ward wrote:

...

On 04 September 2000, M.-A. Lemburg said:

...
...
Correct. MANIFEST is a simple list of files, one per line. If you want to include whole directories, you should write a MANIFEST.in file.

I didn't mean that distutils should scan the goven directory -- it should simply ignore it instead of raising an exception.

Hmm, are you really getting an exception? When I put a directory name into a MANIFEST file, I get:

error: can't copy 'pkg': doesn't exist or not a regular file

and immediate termination of the setup script (hey, it's an error, not a warning). I'll admit that's not as useful as it could be, but as long as there's no traceback it's not a bug! ;-)

You're right; it's not an exception, but still something which causes distutils to stop ;-) Would it hurt much adding a "continue" somewhere to remedy this minor "caveat" :-)

...

Actually, I spent some time documenting the "sdist" command today, and tracked down some bugs in the handling of the manifest files. I also now understand the rules a bit better -- amazing what reading the code will tell you. ;-) Here are the rules as implemented in the current code (not even checked in yet, because of the 2.0b1 code freeze):

cases: 1) no manifest, template exists: generate manifest (covered by 2a: no manifest == template newer) 2) manifest & template exist: 2a) template or setup script newer than manifest: regenerate manifest 2b) manifest newer than both: do nothing (unless --force or --manifest-only) 3) manifest exists, no template: do nothing (unless --force or --manifest-only) 4) no manifest, no template: generate w/ warning ("defaults only")

In the 0.9.2 code (and all Distutils releases since at least 0.2), case (4) wasn't handled properly. A workaround is to use the -f (--force-manifest) option, which causes the sdist command to unconditionally regenerate the MANIFEST file.

Also, case (3) implies that the "default file set" -- setup.py, README.txt, and any source code specified in the setup script -- will *not* be used if you supply your own MANIFEST file. Additionally, the default "prune list" -- which strips RCS and CVS directories, as well as Distutils-generated temporary directories -- does not apply. IOW, if you generate your own MANIFEST, you have to get it exactly right. I think this is the right thing to do, because only control freaks will want to supply their own MANIFEST file, and taking away any control from control freaks makes them, well, freak out. ;-)

True ;-)

...

See the 'get_file_list()' method in distutils/command/sdist.py if you're curious about the implementation. (And see if you can spot the bug that I fixed this morning.) (No fair watching python-checkins!)

...
I'm currently using my own tools for generating the MANIFEST file. The most important difference is that they allow per directory MANIFEST.in style files which are appended to the general MANIFEST.in logic while scanning the directory.

Hmmm, another good reason to roll your own manifest -- maybe it's not just for control freaks?

...
My version if rpm 3.0.3. [...] I also noted another bug: When building an RPM which contains more than one Extension(), bdist_rpm fails on the second Extension(): it can't find the C file. """ running build_ext building 'mx.DateTime.mxDateTime.mxDateTime' extension creating build/temp.linux2 creating build/temp.linux2/mx creating build/temp.linux2/mx/DateTime creating build/temp.linux2/mx/DateTime/mxDateTime gcc -g -O2 -fpic -Imx/DateTime/mxDateTime -I/usr/local/include/python2.0 -c mx/DateTime/mxDateTime/mxDateTime.c -o build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -O2 -m486 -fno-strength-reduce creating build/lib.linux2/mx/DateTime creating build/lib.linux2/mx/DateTime/mxDateTime gcc -shared build/temp.linux2/mx/DateTime/mxDateTime/mxDateTime.o -o build/lib.linux2/mx/DateTime/mxDateTime/mxDateTime.so building 'mx.Proxy.mxProxy.mxProxy' extension error: file 'mx/Proxy/mxProxy/mxProxy.c' does not exist Bad exit status from /var/tmp/rpm-tmp.85854 (%build) error: command 'rpm' failed with exit status 1 """

??? Very weird. Does "python setup.py build" work on its own, ie. when run by you rather than by rpm in its build directory?

Yes. Works just fine.

...

One simple possibility is that you forgot to include mx/Proxy/mxProxy/mxProxy.c in your tarball -- make sure that you can unpack your tarball in a fresh directory and run "python setup.py build", which is just what rpm is doing.

That could have been the cause. While testing this I stumbled over another bugglet: Executing: %build + umask 022 + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD + cd mx-Extensions-BASE-1.0.0 + env 'CFLAGS=-O2 -m486 -fno-strength-reduce' python setup.py build Traceback (innermost last): File "setup.py", line 7, in ? from distutils.core import setup, Extension File "/home/lemburg/lib/distutils/core.py", line 146 raise ^ SyntaxError: invalid syntax This is due to RPM finding a Python 1.5 version under the simple name "python".... distutils doesn't seem to be compatible with multiple installed Python versions :-( [I did run the setup.py file using Python 2.0]. I then tried: projects/tmp> python2.0 setup.py bdist_rpm --prep-script ./prep-script invalid command name './prep-script' Looks like somethings wrong there: I can't seem to pass a script filename to the command. I had to hack the bdist_rpm.py file to make it work (perhaps time for yet another option ?-) ... OK, after that hack the RPM compile runs through, but it now finishes with the following lines (and no apparent reason): creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools copying build/lib.linux2/mx/Tools/mxTools/mxTools.so -> /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools byte-compiling /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/__init__.py to __init__.pyc writing list of installed files to 'INSTALLED_FILES' warning: install: modules installed to '/var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself + exit 0 Processing files: mx-Extensions-BASE File listed twice: /usr/local/lib/python2.0/site-packages/mx/Tools/mxTools/mxTools.so Finding provides... Finding requires... Provides: mxDateTime.so mxProxy.so mxQueue.so mxStack.so mxTextTools.so mxTools.so Requires: ld-linux.so.2 libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) error: command 'rpm' failed with exit status 1 Does this mean that my system doesn't provide the "required" packages ? (SuSE has a somewhat different naming scheme than RedHat so this could be a possible cause)

...

...
...
For a straight Distutil build (ie. no RPM involved), setting CFLAGS should work, but it only applies to extensions (not C libraries). When RPM is involved, I'm not sure what the Right Way to do it is. Maybe CFLAGS will work? Maybe editing /etc/rpmrc or /usr/lib/rpm/rpmrc? I dunno.

Wouldn't an option to distutils be the right way for these kind of things ?

Yes. Definitely. It's on the TODO list.

Great :-)

...

Now, on to your other "bdist_rpm" failure...

...
setup.py bdist_rpm output:

rpm -ba --define _topdir /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm --clean build/bdist.linux2/rpm/SPECS/mx-Extensions-BASE.spec Executing: %prep [...] Wrote: /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/SRPMS/mx-Extensions-BASE-1.0.0-1.src.rpm Could not open /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386/mx-Extensions-BASE-1.0.0-1.i386.rpm

error: command 'rpm' failed with exit status 1

Hmmm, this smells like the archetypal Distutils bug: someone's trying to write a file into a directory that doesn't exist yet. In this case, it would be rpm's fault, since the "bdist_rpm" command doesn't know the name of the architecture directory that rpm will try to write to. Things to look for: * does /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386 exist?

No.

...

* if you create it, does the bdist_rpm command then run successfully?

Yes.

...

I haven't had problems with rpm failing to create directories -- I have RPM 3.0.4 on my Red Hat 6.2 box, and I just cranked out an RPM of mxDateTime 1.3.0 without a hitch. Perhaps RPM 3.0.3 joins RPM 2.x on the index of forbidden versions... sigh...

Dunno. It did work after I manually created the i386 directory and applied the python2.0 hack to bdist_rpm.py. Thanks, -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Greg Ward

3:47 p.m.

On 04 September 2000, M.-A. Lemburg said:

...

...
error: can't copy 'pkg': doesn't exist or not a regular file

and immediate termination of the setup script (hey, it's an error, not a warning). I'll admit that's not as useful as it could be, but as long as there's no traceback it's not a bug! ;-)

You're right; it's not an exception, but still something which causes distutils to stop ;-)

It is fixable, I was just making sure that you and I were seeing the same problem -- "that's not a bug, it's a misfeature". Another thing I discovered this morning is that if MANIFEST is empty, then you *do* get an exception -- definitely a bug. So at least a "! empty" check of the file list is required before trying to copy files and create the tarball; might as well throw in a "isfile or warn" check on everything at the same time.

...

Would it hurt much adding a "continue" somewhere to remedy this minor "caveat" :-)

Well, it'll print a warning that "foo is not a regular file (skipped)". Not nice to silently skip things!

...

That could have been the cause. While testing this I stumbled over another bugglet:

Executing: %build + umask 022 + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD + cd mx-Extensions-BASE-1.0.0 + env 'CFLAGS=-O2 -m486 -fno-strength-reduce' python setup.py build Traceback (innermost last): File "setup.py", line 7, in ? from distutils.core import setup, Extension File "/home/lemburg/lib/distutils/core.py", line 146 raise ^ SyntaxError: invalid syntax

This is due to RPM finding a Python 1.5 version under the simple name "python".... distutils doesn't seem to be compatible with multiple installed Python versions :-( [I did run the setup.py file using Python 2.0].

Yup, there's another thread about this very issue going on at this very moment. Everyone who has spoken up (me, AMK, and Harry Gebel) thinks that it should at least be an option to put the value of sys.executable in the spec file, instead of hard-coding "python". I think you've raised another good argument in favour of making this the default, but I'm biased -- I do think it should be the default. ;-)

...

I then tried:

projects/tmp> python2.0 setup.py bdist_rpm --prep-script ./prep-script invalid command name './prep-script'

Looks like somethings wrong there: I can't seem to pass a script filename to the command.

The RPM script options are, ummm, not well-thought-out. Possibly unfinished, I'm not sure. Those are the options I was thinking of when I said it might have been unwise to attempt to mirror *all* of RPM's spec file in Distutils options. My inclination is to delete them, rather than expose a broken interface.

...

OK, after that hack the RPM compile runs through, but it now finishes with the following lines (and no apparent reason):

creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools copying build/lib.linux2/mx/Tools/mxTools/mxTools.so -> /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools byte-compiling /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/__init__.py to __init__.pyc writing list of installed files to 'INSTALLED_FILES' warning: install: modules installed to '/var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself + exit 0 Processing files: mx-Extensions-BASE File listed twice: /usr/local/lib/python2.0/site-packages/mx/Tools/mxTools/mxTools.so Finding provides... Finding requires... Provides: mxDateTime.so mxProxy.so mxQueue.so mxStack.so mxTextTools.so mxTools.so Requires: ld-linux.so.2 libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) error: command 'rpm' failed with exit status 1

??? I'm confused again. Try running "bdist_rpm" with the --source-only option, and then see if you can build from the source RPM directly. About the only way to debug these sort of problems is to break the thing down, and a *heck* of a lot of stuff happens between you typing "setup.py bdist_rpm" and the finished RPM files being moved into dist/. (Just listen to your hard drive go!)

...

...
* does /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386 exist?

No.

...
* if you create it, does the bdist_rpm command then run successfully?

Yes.

OK, based on that and Andrew's feedback, I'd say we blame RPM 3.0.3. Bottom line: building RPMs with the Distutils requires RPM 3.0.4 or better. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/

M.-A. Lemburg

8:55 p.m.

Greg Ward wrote:

...

On 04 September 2000, M.-A. Lemburg said:

...
...
error: can't copy 'pkg': doesn't exist or not a regular file

and immediate termination of the setup script (hey, it's an error, not a warning). I'll admit that's not as useful as it could be, but as long as there's no traceback it's not a bug! ;-)

You're right; it's not an exception, but still something which causes distutils to stop ;-)

It is fixable, I was just making sure that you and I were seeing the same problem -- "that's not a bug, it's a misfeature". Another thing I discovered this morning is that if MANIFEST is empty, then you *do* get an exception -- definitely a bug. So at least a "! empty" check of the file list is required before trying to copy files and create the tarball; might as well throw in a "isfile or warn" check on everything at the same time.

...
Would it hurt much adding a "continue" somewhere to remedy this minor "caveat" :-)

Well, it'll print a warning that "foo is not a regular file (skipped)". Not nice to silently skip things!

Ok.

...

...
That could have been the cause. While testing this I stumbled over another bugglet:

Executing: %build + umask 022 + cd /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/BUILD + cd mx-Extensions-BASE-1.0.0 + env 'CFLAGS=-O2 -m486 -fno-strength-reduce' python setup.py build Traceback (innermost last): File "setup.py", line 7, in ? from distutils.core import setup, Extension File "/home/lemburg/lib/distutils/core.py", line 146 raise ^ SyntaxError: invalid syntax

This is due to RPM finding a Python 1.5 version under the simple name "python".... distutils doesn't seem to be compatible with multiple installed Python versions :-( [I did run the setup.py file using Python 2.0].

Yup, there's another thread about this very issue going on at this very moment. Everyone who has spoken up (me, AMK, and Harry Gebel) thinks that it should at least be an option to put the value of sys.executable in the spec file, instead of hard-coding "python". I think you've raised another good argument in favour of making this the default, but I'm biased -- I do think it should be the default. ;-)

Hmm, this depends on how the spec file is created. [Warning: I don't know much about RPM...] If the contents of the spec are included in the RPM file then it is probably a bad idea to include sys.executable of the building Python interpreter. In that case, I'd leave the hard- coded "python" in there and issue a warning that the executable found as "python" is not identical to sys.executable. The reason is simple: you cannot know what Python version a user uses to rebuild from SRPM. If the spec is *always* recreated, then I'd opt for the sys.executable default too, since it is natural that the Python version running setup.py will also take care of building the RPM sources.

...

...
I then tried:

projects/tmp> python2.0 setup.py bdist_rpm --prep-script ./prep-script invalid command name './prep-script'

Looks like somethings wrong there: I can't seem to pass a script filename to the command.

The RPM script options are, ummm, not well-thought-out. Possibly unfinished, I'm not sure. Those are the options I was thinking of when I said it might have been unwise to attempt to mirror *all* of RPM's spec file in Distutils options. My inclination is to delete them, rather than expose a broken interface.

Fine with me... as long as you do publish a working interface to pass arbitrary RPM options to "rpm" ;-)

...

...
OK, after that hack the RPM compile runs through, but it now finishes with the following lines (and no apparent reason):

creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools copying build/lib.linux2/mx/Tools/mxTools/mxTools.so -> /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools byte-compiling /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/__init__.py to __init__.pyc writing list of installed files to 'INSTALLED_FILES' warning: install: modules installed to '/var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself + exit 0 Processing files: mx-Extensions-BASE File listed twice: /usr/local/lib/python2.0/site-packages/mx/Tools/mxTools/mxTools.so Finding provides... Finding requires... Provides: mxDateTime.so mxProxy.so mxQueue.so mxStack.so mxTextTools.so mxTools.so Requires: ld-linux.so.2 libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) error: command 'rpm' failed with exit status 1

??? I'm confused again.

Try running "bdist_rpm" with the --source-only option, and then see if you can build from the source RPM directly. About the only way to debug these sort of problems is to break the thing down, and a *heck* of a lot of stuff happens between you typing "setup.py bdist_rpm" and the finished RPM files being moved into dist/. (Just listen to your hard drive go!)

This works. Hmm, runnning /home/lemburg> rpm -q --whatprovides ld-linux.so.2 libc.so.6 "libc.so.6(GLIBC_2.0)" "libc.so.6(GLIBC_2.1)" I get: shlibs-2.1.1-4 shlibs-2.1.1-4 javarunt-1.1.7v1a-11 shlibs-2.1.1-4 shlibs-2.1.1-4 This looks ok, so it's not the "Requires" part which causes the problem either.

...

...
...
* does /data/home/lemburg/projects/tmp/build/bdist.linux2/rpm/RPMS/i386 exist?

No.

...
* if you create it, does the bdist_rpm command then run successfully?

Yes.

OK, based on that and Andrew's feedback, I'd say we blame RPM 3.0.3.

Bottom line: building RPMs with the Distutils requires RPM 3.0.4 or better.

I'll try upgrading to rpm 3.0.4... -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Harry Henry Gebel

9:04 p.m.

On Tue, Sep 05, 2000 at 09:54:24AM +0200, M.-A. Lemburg wrote:

...

If the contents of the spec are included in the RPM file then it is probably a bad idea to include sys.executable of the building Python interpreter. In that case, I'd leave the hard- coded "python" in there and issue a warning that the executable found as "python" is not identical to sys.executable. The reason is simple: you cannot know what Python version a user uses to rebuild from SRPM.

If the spec is *always* recreated, then I'd opt for the sys.executable default too, since it is natural that the Python version running setup.py will also take care of building the RPM sources.

The contents of the spec file are included in the source RPM, that is why A think that using sys.executable should be the option and not the default. -- Harry Henry Gebel, Senior Developer, Landon House SBS ICQ# 76308382 West Dover Hundred, Delaware

M.-A. Lemburg

9:39 p.m.

"M.-A. Lemburg" wrote:

...

...
...
OK, after that hack the RPM compile runs through, but it now finishes with the following lines (and no apparent reason):

creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools creating /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools copying build/lib.linux2/mx/Tools/mxTools/mxTools.so -> /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/Tools/mxTools byte-compiling /var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/mx/__init__.py to __init__.pyc writing list of installed files to 'INSTALLED_FILES' warning: install: modules installed to '/var/tmp/mx-Extensions-BASE-buildroot/usr/local/lib/python2.0/site-packages/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself + exit 0 Processing files: mx-Extensions-BASE File listed twice: /usr/local/lib/python2.0/site-packages/mx/Tools/mxTools/mxTools.so Finding provides... Finding requires... Provides: mxDateTime.so mxProxy.so mxQueue.so mxStack.so mxTextTools.so mxTools.so Requires: ld-linux.so.2 libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) error: command 'rpm' failed with exit status 1

??? I'm confused again.

Try running "bdist_rpm" with the --source-only option, and then see if you can build from the source RPM directly. About the only way to debug these sort of problems is to break the thing down, and a *heck* of a lot of stuff happens between you typing "setup.py bdist_rpm" and the finished RPM files being moved into dist/. (Just listen to your hard drive go!)

This works.

Hmm, runnning

/home/lemburg> rpm -q --whatprovides ld-linux.so.2 libc.so.6 "libc.so.6(GLIBC_2.0)" "libc.so.6(GLIBC_2.1)"

I get:

shlibs-2.1.1-4 shlibs-2.1.1-4 javarunt-1.1.7v1a-11 shlibs-2.1.1-4 shlibs-2.1.1-4

This looks ok, so it's not the "Requires" part which causes the problem either.

Ok, I found the bug... RPM didn't like a mistake I made in the setup.py file: I listed one Extension twice. While distutils didn't warn about this, RPM fails to produce a binary RPM if a file is "listed twice". Removing the duplicate in setup.py remedies the problem. Now I have another problem ;-): the created RPM doesn't include all needed files, just the base package's __init__.py file and the compiled .so files: tmp/dist> rpm -qpl mx-Extensions-BASE-1.0.0-1.i386.rpm /usr/local/lib/python2.0/site-packages/mx/DateTime/mxDateTime/mxDateTime.so /usr/local/lib/python2.0/site-packages/mx/Proxy/mxProxy/mxProxy.so /usr/local/lib/python2.0/site-packages/mx/Queue/mxQueue/mxQueue.so /usr/local/lib/python2.0/site-packages/mx/Stack/mxStack/mxStack.so /usr/local/lib/python2.0/site-packages/mx/TextTools/mxTextTools/mxTextTools.so /usr/local/lib/python2.0/site-packages/mx/Tools/mxTools/mxTools.so /usr/local/lib/python2.0/site-packages/mx/__init__.py /usr/local/lib/python2.0/site-packages/mx/__init__.pyc Here's my setup.py file: #!/usr/bin/env python2.0 """ Distutils Setup File for the mx Extensions BASE distribution. """ import string from distutils.core import setup, Extension from distutils.command.config import config __version__ = '1.0.0' setup (name = "mx-Extensions-BASE", version = __version__, description = "", long_description = "", author = "Marc-André Lemburg", author_email = "mal@lemburg.com", url = "http://starship.python.net/~lemburg/mxBASE.html", packages = ['mx'], ext_modules = [ Extension('mx.DateTime.mxDateTime.mxDateTime', ['mx/DateTime/mxDateTime/mxDateTime.c'], include_dirs=['mx/DateTime/mxDateTime']), Extension('mx.Proxy.mxProxy.mxProxy', ['mx/Proxy/mxProxy/mxProxy.c'], include_dirs=['mx/Proxy/mxProxy']), Extension('mx.Queue.mxQueue.mxQueue', ['mx/Queue/mxQueue/mxQueue.c'], include_dirs=['mx/Queue/mxQueue']), Extension('mx.Stack.mxStack.mxStack', ['mx/Stack/mxStack/mxStack.c'], include_dirs=['mx/Stack/mxStack']), Extension('mx.TextTools.mxTextTools.mxTextTools', ['mx/TextTools/mxTextTools/mxTextTools.c'], include_dirs=['mx/TextTools/mxTextTools']), Extension('mx.Tools.mxTools.mxTools', ['mx/Tools/mxTools/mxTools.c'], include_dirs=['mx/Tools/mxTools']), ] ) Do I need to include *all* subpackages in the packages list ? (distutils seems to work without it.) -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Greg Ward

9 Sep 9 Sep

5:32 a.m.

On 05 September 2000, Harry Henry Gebel said:

...

The contents of the spec file are included in the source RPM, that is why A think that using sys.executable should be the option and not the default.

Dammit, I'm still waiting for a light to flash on so I can understand where you guys are coming from on this. Please indulge me as I think out loud... Assertions (please tell me if I'm misunderstanding something): * this isn't really all *that* important, as the build instructions in the .spec file only apply to people building the source RPM -- IOW, the build instructions in the .spec file in no way affect the majority of people who use the RPM, ie. those who just install the "built" RPM * in fact, the only person who *must* build the source RPM is the person who creates the RPM in the first place (although anyone creating binary RPMs for other distributions or other architectures would probably start from the source RPM) I've been thinking hard about this -- hey, my brain is slow this weekend -- and I think I understand it a little better. Case 1 (the status quo): put "python setup.py ..." in the build/install/ clean instructions in the .spec file. This is bad when the packager, P, uses a "non-standard" python (anything other than /usr/bin/python) to create an RPM that is intended to go in the non-standard location. This is mainly a problem when P immediately creates a built RPM from his new source RPM (the usual case, in fact); if a builder, B, turns source RPM -> built RPM on a separate system, then using the first python in the path -- most likely /usr/bin/python -- might well be the right thing to do. Case 2: put sys.executable + " setup.py ..." in the .spec file. This fixes the above problem, but is bad in the case where P accidentally uses a non-standard python to create an RPM that is supposed to go to the standard python installation (/usr). Eg. if I forget that /usr/local/bin/python is first in my path, then any source RPMs I create will refer to /usr/local/bin/python in the .spec file, and building those source RPMs will either fail (on systems that don't have /usr/local/bin/python, probably the vast majority of installed Linux boxen out there) or will generate an RPM with the "non-standard" destination of /usr/local. In the latter case -- accidentally using the wrong python -- it would be best if the Distutil issued a warning. I don't see an easy way to detect this, especially if someone is deliberately creating an RPM to install modules to a non-standard location. (Eg. Andrew's case of having python in /www/python/bin/python: he might want to make RPMs of common modules to install on all the web developers' workstations in /www/python.) Both situations are subtle errors on the packager's part, and neither seem to have obvious automatic solutions. The "fix" is social engineering: let the packager decide what he wants to do with options to the bdist_rpm command, and make sure the rationale for each option is carefully documented. The question is: which failure is more obvious and happens closest to the packager, the one who made the mistake (failed to RTFM, etc.)? That one should be the default. It seems to me that creating a source RPM (spec file) that immediately fails because of an explicit /path/to/python is better than one that works, but possibly wrongly, because of implicitly using the first python on the path. IOW, "Explicit is better than implicit", even in snippets of shell code included in a .spec file bundled in a source RPM. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/ "I came, I saw, she conquered." (The original Latin seems to have been garbled.)

Greg Ward

5:34 a.m.

On 05 September 2000, Harry Henry Gebel said:

...

The contents of the spec file are included in the source RPM, that is why A think that using sys.executable should be the option and not the default.

But it only affects people building from the source RPM, which is usually done by the same person as created the source RPM and on the same system. See my last post for details. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/

M.-A. Lemburg

6:05 a.m.

Greg Ward wrote:

...

On 05 September 2000, Harry Henry Gebel said:

...
The contents of the spec file are included in the source RPM, that is why A think that using sys.executable should be the option and not the default.

Dammit, I'm still waiting for a light to flash on so I can understand where you guys are coming from on this. Please indulge me as I think out loud...

Assertions (please tell me if I'm misunderstanding something):

* this isn't really all *that* important, as the build instructions in the .spec file only apply to people building the source RPM -- IOW, the build instructions in the .spec file in no way affect the majority of people who use the RPM, ie. those who just install the "built" RPM

* in fact, the only person who *must* build the source RPM is the person who creates the RPM in the first place (although anyone creating binary RPMs for other distributions or other architectures would probably start from the source RPM)

I've been thinking hard about this -- hey, my brain is slow this weekend -- and I think I understand it a little better.

Case 1 (the status quo): put "python setup.py ..." in the build/install/ clean instructions in the .spec file. This is bad when the packager, P, uses a "non-standard" python (anything other than /usr/bin/python) to create an RPM that is intended to go in the non-standard location. This is mainly a problem when P immediately creates a built RPM from his new source RPM (the usual case, in fact); if a builder, B, turns source RPM -> built RPM on a separate system, then using the first python in the path -- most likely /usr/bin/python -- might well be the right thing to do.

Case 2: put sys.executable + " setup.py ..." in the .spec file. This fixes the above problem, but is bad in the case where P accidentally uses a non-standard python to create an RPM that is supposed to go to the standard python installation (/usr). Eg. if I forget that /usr/local/bin/python is first in my path, then any source RPMs I create will refer to /usr/local/bin/python in the .spec file, and building those source RPMs will either fail (on systems that don't have /usr/local/bin/python, probably the vast majority of installed Linux boxen out there) or will generate an RPM with the "non-standard" destination of /usr/local.

In the latter case -- accidentally using the wrong python -- it would be best if the Distutil issued a warning. I don't see an easy way to detect this, especially if someone is deliberately creating an RPM to install modules to a non-standard location. (Eg. Andrew's case of having python in /www/python/bin/python: he might want to make RPMs of common modules to install on all the web developers' workstations in /www/python.)

Both situations are subtle errors on the packager's part, and neither seem to have obvious automatic solutions. The "fix" is social engineering: let the packager decide what he wants to do with options to the bdist_rpm command, and make sure the rationale for each option is carefully documented.

The question is: which failure is more obvious and happens closest to the packager, the one who made the mistake (failed to RTFM, etc.)? That one should be the default. It seems to me that creating a source RPM (spec file) that immediately fails because of an explicit /path/to/python is better than one that works, but possibly wrongly, because of implicitly using the first python on the path.

IOW, "Explicit is better than implicit", even in snippets of shell code included in a .spec file bundled in a source RPM.

Why all the fuzz ? ;-) Keep the status quo and add an option for those who want to play with different Python versions, e.g. --set-executable=/home/lemburg/tmp/betas/python2.0 This will make everybody happy and hide the feature well enough to not cause frequent RTFM replies. Those who need it will find it and those who can't find it won't need it ;-) KISS, -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Mark W. Alexander

1:25 p.m.

On Sat, 9 Sep 2000, Greg Ward wrote:

...

On 05 September 2000, Harry Henry Gebel said:

...
The contents of the spec file are included in the source RPM, that is why A think that using sys.executable should be the option and not the default.

Dammit, I'm still waiting for a light to flash on so I can understand where you guys are coming from on this. Please indulge me as I think out loud...

Making packages is not that hard. Making GOOD packages is very subtly tricky. My comments below may help your understanding. They may not help find a solution....

...

Assertions (please tell me if I'm misunderstanding something):

* this isn't really all *that* important, as the build instructions in the .spec file only apply to people building the source RPM -- IOW, the build instructions in the .spec file in no way affect the majority of people who use the RPM, ie. those who just install the "built" RPM

Correct.

...

* in fact, the only person who *must* build the source RPM is the person who creates the RPM in the first place (although anyone creating binary RPMs for other distributions or other architectures would probably start from the source RPM)

Not exactly. No one ever *must* build a source RPM. If you have a .spec and a source tar/directory you can build a binary RPM without creating a source RPM. Distribution of the binary RPM satisfies 99.99% of the people who want to install an RPM. Those who want to tweak how the binary is created could use the source RPM, but even then (especially with pure python packages) it's not required. I've "rebuilt" RPM's by getting a source tarball and recreating most of the .spec from an rpm -qi query.

...

I've been thinking hard about this -- hey, my brain is slow this weekend -- and I think I understand it a little better.

Case 1 (the status quo): put "python setup.py ..." in the build/install/ clean instructions in the .spec file. This is bad when the packager, P, uses a "non-standard" python (anything other than /usr/bin/python) to create an RPM that is intended to go in the non-standard location. This is mainly a problem when P immediately creates a built RPM from his new source RPM (the usual case, in fact); if a builder, B, turns source RPM -> built RPM on a separate system, then using the first python in the path -- most likely /usr/bin/python -- might well be the right thing to do.

Close. It really has nothing to do with the source RPM. It's just the usual RPM way of package building. I prefer to "force" my rpm installs (the fake install done by rpm -b during the install step) into a directory OTHER than where it will install on the target machine. Benefits are 1) I can create a binary RPM on a system where I don't hace root access (as if I'd work there...), and 2) you don't step on your production files untill you are satisfied with the RPM. This has the added advantage in that you can now make the RPM relocatable with a little more effort in the .spec file. To make the package relocatable, you capture the library path where you want it to go (presumably the site-packages directory of the default python, but it doesn't have to be) to the location where you just did the fake RPM install. RPM then works it's magic by creating the package from the files in the fake tree so that they will be installed by default into the real site-packages directory. Since the package is relocatable now, the installer could override it to go anywhere. This is really cool, in that now the installer doesn't have to be root. Instead, they can relocate the package anywhere they can write and just add that path to their PYTHONPATH.

...

Case 2: put sys.executable + " setup.py ..." in the .spec file. This fixes the above problem, but is bad in the case where P accidentally uses a non-standard python to create an RPM that is supposed to go to the standard python installation (/usr). Eg. if I forget that /usr/local/bin/python is first in my path, then any source RPMs I create will refer to /usr/local/bin/python in the .spec file, and building those source RPMs will either fail (on systems that don't have /usr/local/bin/python, probably the vast majority of installed Linux boxen out there) or will generate an RPM with the "non-standard" destination of /usr/local.

There's a little more confusion here, and I'm not exactly sure how it works with bdist_rpm. RPM does some additional hacking beyond what's in the .spec file. It looks especially for #! magic in package files but also for referenced libraries and anything else it thinks the package is going to need. RPM then (theoritcally, in a helpful fashion) adds these things to the "Requires:" tag in the binary package. SO, if it identifies /usr/local/bin/python as a requirement for the package, it will not install (without --force) on a target system that does not have /usr/local/bin/python. (Yes, I have done this...the original pygresql package went out this way...sigh). I don't think RPM deduces requires from processors used in the setup, config, build, install steps, but if the packager is using a /usr/local/bin python, it's very likely to be references somewhere in the package, too.

...

In the latter case -- accidentally using the wrong python -- it would be best if the Distutil issued a warning. I don't see an easy way to detect this, especially if someone is deliberately creating an RPM to install modules to a non-standard location. (Eg. Andrew's case of having python in /www/python/bin/python: he might want to make RPMs of common modules to install on all the web developers' workstations in /www/python.)

This is why I'm struggling with pkgtool (and even worse, sdux). If the bdist_whatever's make the package relocatable, it doesn't matter what python was used in the build (location-wise anyway). Then the issue shift's to installation time. How do you determin the TARGET system's "prefered" python? This could be retrieved either by executing python or querying RPM in a preinstall step, then automatically reloacting the package where it should go. Except you also have to be prepared to skip the auto-relocation if the installer is relocating it with RPM options.

...

Both situations are subtle errors on the packager's part, and neither seem to have obvious automatic solutions. The "fix" is social engineering: let the packager decide what he wants to do with options to the bdist_rpm command, and make sure the rationale for each option is carefully documented.

Like I said...Packaging is easy. Making easy packages are hard. I'd vote for making bdist_rpm make things relocatable with a default pre-install to do auto-relocation IF I still had any sanity from actually trying to do this with other package types. Maybe I'm too anal about my packages, but good ones make it sooooo much easier to administer large number of machines.

...

The question is: which failure is more obvious and happens closest to the packager, the one who made the mistake (failed to RTFM, etc.)? That one should be the default. It seems to me that creating a source RPM (spec file) that immediately fails because of an explicit /path/to/python is better than one that works, but possibly wrongly, because of implicitly using the first python on the path.

Agreed, except that the source RPM is still only going to be used by people who are already prepared to get in and hack at it anyway.

...

IOW, "Explicit is better than implicit", even in snippets of shell code included in a .spec file bundled in a source RPM.

Hey, you're not trying to suggest that maybe, possible, ONE thing from bdist_pkgtool was almost, kinda, maybe on the right track ;-) Anyway I said this probably wouldn't be a solution, but I hope I clarified the problem al little.... mwa

Greg Ward

1:59 p.m.

On 09 September 2000, M.-A. Lemburg said:

...

Why all the fuzz ? ;-)

'cause I spent waaaay too much time Friday night and Saturday morning worrying about this extremely trivial issue, and I wanted the rest of the SIG to share my pain. So there!

...

Keep the status quo and add an option for those who want to play with different Python versions, e.g. --set-executable=/home/lemburg/tmp/betas/python2.0

Done. Here's the help text from "bdist_rpm --help": --python path to Python interpreter to hard-code in the .spec file (default: "python") --fix-python hard-code the exact path to the current Python interpreter in the .spec file Seem reasonable? There's no opposite to --fix-python, because you can go back to the default with "--python=python". Should be checked in by the time you read this... Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/

8623

Age (days ago)

8630

Last active (days ago)

List overview

Download

17 comments

7 participants

participants (7)

Andrew Kuchling
Greg Ward
Harry Henry Gebel
Harry Henry Gebel
M.-A. Lemburg
Mark W. Alexander
Thomas Heller

First steps with distutils...

M.-A. Lemburg

Greg Ward

Harry Henry Gebel

M.-A. Lemburg

M.-A. Lemburg

Thomas Heller

Greg Ward

Andrew Kuchling

M.-A. Lemburg

Greg Ward

M.-A. Lemburg

Harry Henry Gebel

M.-A. Lemburg

Greg Ward

Greg Ward

M.-A. Lemburg

Mark W. Alexander

Greg Ward

tags

participants (7)