Mailman 3 May 2000 - Distutils-SIG

Parsing config files + passing options
by Brad Chapman 11 May '00

11 May '00

Hello all! I've been playing around with Distutils-0.8.2 and just have a few questions. Before I start though, thanks for all your work on this so far, it is *very* nice and much needed! I just have two questions: 1. I'm working on writing a setup.py for an extension that uses gnome and glib libraries, and was curious if there was a general way anyone has already thought of to handle getting information from gnome-config and glib-config calls into a form that you can input into Distutils. For instance, if I do something like the following: >>> import commands >>> gthread_cflags = commands.getoutput("glib-config --cflags gthread") >>> print gthread_cflags -I/usr/local/include/glib12 -D_THREAD_SAFE I can get the options you would pass. It would be useful to have something to process this like: def parse_cflag_config(configure_script, library) which would then return two lists, the include dirs and the macros. So they you could do something in your setup.py like: gthread_includes, gthread_macros = parse_cflag_config("glib-config", "gthread") Of course, this would only work on UNIX, but that is where the config calls are anyways :-) Is this something that would be generally useful to anyone? Does this approach seem like a good idea? Is there already a better plan in place to handle this? 2. Going along with this first problem, I was also wondering how to pass specific compiler flags. The issue I have is that on FreeBSD, there is a FreeBSD specific option for linking the thread libraries: '-pthread' (what this option does is link in libc_r and not libc, if anyone cares. I'm afraid I'm not smart enough to more about it than that :-). Anyways, is there a way to pass a flag like '-pthread'? Apologies if I missed it! Thanks much for listening and thanks again for the wonderful tool. Brad

2 1

Thoughts about Distutils and Debian
by Gregor Hoffleit 11 May '00

11 May '00

Hi, I'm currently trying to make up my mind how Distutils and building Debian packages fit together. I have to impression we're faced with something like a chicken and egg problem. ... Since not everybody knows the Debian package format, I'll give a short introduction: A short story of .deb's and Debian packages ------------------------------------------- A .deb package is in fact really, really simple: freefly;158> ar tv python-ldap_1.8-1_i386.deb rw-r--r-- 0/0 4 Nov 30 15:58 1999 debian-binary rw-r--r-- 0/0 1078 Nov 30 15:58 1999 control.tar.gz rw-r--r-- 0/0 52562 Nov 30 15:58 1999 data.tar.gz freefly;159> ar x python-ldap_1.8-1_i386.deb freefly;160> cat debian-binary 2.0 freefly;161> tar tvpzf control.tar.gz drwxr-xr-x root/root 0 1999-11-30 15:58:41 ./ -rwxr-xr-x root/root 553 1999-11-30 15:58:05 ./postinst -rwxr-xr-x root/root 461 1999-11-30 15:58:05 ./prerm -rw-r--r-- root/root 660 1999-11-30 15:58:41 ./md5sums -rw-r--r-- root/root 392 1999-11-30 15:58:41 ./control freefly;162> tar tvpzf data.tar.gz drwxr-xr-x root/root 0 1999-11-30 15:58:04 ./ drwxr-xr-x root/root 0 1999-11-30 15:58:01 ./usr/ drwxr-xr-x root/root 0 1999-11-30 15:58:01 ./usr/lib/ drwxr-xr-x root/root 0 1999-11-30 15:58:01 ./usr/lib/python1.5/ ... -rw-r--r-- root/root 228 1999-11-30 15:48:07 ./usr/share/doc-base/python-ldap The nice thing is that you can simply unpack the file with standard Unix tools. The same is true for the reverse: You can build a .deb with ar and tar only. What's the stuff in control.tar.gz ? postinst and prerm (and preinst and postrm) are optional scripts that are called at various points of the installation/deinstallation of a package; control is a text file describing the package: freefly;163> cat control Package: python-ldap Version: 1.8-1 Section: net Priority: optional Architecture: i386 Depends: libc6 (>= 2.1), libopenldap1 Installed-Size: 99 Maintainer: Gregor Hoffleit <flight(a)debian.org> Description: An LDAP module for Python. This module provides an Python interface to the LDAP client library (LDAP is the Lightweight Directory Access Protocol). It has been compiled with OpenLDAP. Building a valid .deb package is as simple as echo 2.0 >debian-binary vi control tar -c -z -f control.tar.gz ./control tar -c -z -f data.tar.gz -C build . ar cv my1stdeb-0.0.deb debian-binary control.tar.gz data.tar.gz Now while this is a valid .deb file, there's more to a valid Debian package than this: Packages have to follow some rules in order to (a) not to hose the system were they're unpacked (b) fit into the Debian archive scheme (c) support building from source (d) support auto-building from source These rules are put together in the Debian Policy manual (http://www.debian.org/doc/debian-policy/) and the Debian Packaging manual. The most interesting things in our context are certainly (c) and (d). A valid Debian package should be built from a source package. A source package consists of a tarball of the upstream version (xyz_1.2.orig.tar.gz), a diff file with Debian's changes to the package (xyz_1.2-5.diff.gz) and a text file with a description of the package (xyz_1.2-5.dsc) including things like md5sums of the source files and GPG signed by the maintainer of the package. (If you have these three files, you can unpack the source tree with "dpkg-source -x xyz_1.2-5.dsc", or with tar -xzpf xyz_1.2.orig.tar.gz zcat xyz_1.2-5.diff.gz | patch -p1 Debian's equivalent to an RPM spec file is a ./debian directory in the source tree. Typically, this directory looks like this: freefly;193> cd python-ldap-1.8/ freefly;194> ls -ld debian/* -rw-r--r-- 1 flight flight 323 Mai 10 21:28 debian/README.Debian -rw-r--r-- 1 flight flight 203 Mai 10 21:28 debian/TODO.Debian -rw-r--r-- 1 flight flight 1644 Mai 10 21:28 debian/changelog -rw-r--r-- 1 flight flight 392 Mai 10 21:28 debian/control -rw-r--r-- 1 flight flight 799 Mai 10 21:28 debian/copyright -rw-r--r-- 1 flight flight 228 Mai 10 21:28 debian/doc-base -rw-r--r-- 1 flight flight 133 Mai 10 21:28 debian/postinst -rw-r--r-- 1 flight flight 130 Mai 10 21:28 debian/prerm -rwxr-xr-x 1 flight flight 1576 Mai 10 21:28 debian/rules* The most important thing is debian/rules (note the connotation ;-), which is in fact an (executable) Makefile with targets "build", "clean", "binary-indep", "binary-arch" and "binary". Provided the tools used in this rules Makefile are installed, building all packages from a source tree is done by calling debian/rules binary "debian/rules binary-indep" would build only the packages that are flagged architecture-independent (like a 100% pure Python package); "debian/rules binary-arch" would build the architecture-dependent packages for the architecture of the system we're running on (e.g. i386, m68k, sparc, hurd-i386 or perhaps bsd-i386 or cygwin-i386). For a normal package, "rules build" calls "./configure; make", "rules clean" calls "make clean", "rules binary" calls "make DESTDIR='pwd'/debian/tmp install" and "dh_builddeb" (from a simplified view). Needless to say that Debian's build tools (and autobuild daemon's) rely on the presence of ./debian/rules to build binary packages for every architecture from their source packages. Enters Distutils ---------------- So where exactly does Distutils fit into this picture, how much sense makes bdist_deb, and what should it do ? From the theory above, you see that it's possible and even quite easy to write a bdist_deb target that builds a valid .deb file (using ar and tar) for any Distutil'ed package. The problem is that that's still only a third-class Debian package: Fine for rapid-prototyping of .deb packages, but without support for Debian's infrastructure. If we'd like to do better, we have to support debian/rules somehow. I'm still not sure what's the best way to do this, and how bdist's "source distributions" fit into this. Possible ideas: bdist_deb could create a source archive with a debian/rules skeleton that just works(tm). The debian/rules skeleton would just call up setup.py in different places. I.e. bdist_deb source distribution command creates a source package; this source package uses (in debian/rules) simple setup.py targets like "clean", "build" and "install" and finally builds the .deb file using standard Debian tools. In an ideal world (any Distutil'ed package maps perfectly in a Debian package) this would be fine. In our world, Debian maintainers still would have to merge their changes with new upstream versions, therefore providing a debian/rules skeleton is perhaps not enough. Maybe the bdist_deb binary distribution command then could simply call up "debian/rules binary" to build the binary packages. A little bit more elaborated thoughts to come, I hope. Gregor

1 0

bdist_rpm patch: new package_data
by Harry Henry Gebel 11 May '00

11 May '00

Here is a new bdist_rpm patch that adds the following variables to package_data: Strings: icon - path (relative to setup.py) to an icon which will be used for the package in graphical rpm front-ends distribution_name - placed in spec file 'Distribution' field pre, post, preun, postun, prep, build, install, clean - spec file scripts (defaults are still provided for prep, build, clean, and install that should work 99% of the time) String or number: serial - placed in spec file 'Serial' field Lists of strings or strings: provides, requires, build_requires, conflicts, obsoletes - placed in appropriate fields in spec file The default value of 'group' in package_data has been changed from 'Applications' to 'Development/Libraries'. Type checking has been cleaned up a little. The default addition of 'README' and 'README.txt' to the doc files can be overridden by defining doc as a string rather than as a list of strings. Defining doc as '' will prevent any files including 'README' and 'README.txt' from being include in the doc files. I modified clean.py so that 'clean -a' removes the 'build/rpm' directory. (The rpm program handles cleaning of the 'build/rpm/BUILD' directory, unless 'bdist_rpm --no-clean' is specified. (Or should --no-clean be the default for bdist_rpm?)) -- Harry Henry Gebel, Senior Developer, Landon House SBS West Dover Hundred, Delaware PyNcurses ncurses binding for Python http://pyncurses.sourceforge.net

1 0

Re: [Distutils] Where to find config files
by Michael Muller 10 May '00

10 May '00

Greg Ward wrote: [snip] > On Unix, there are three possible config files: pydistutils.cfg in > the Distutils installation directory (ie. where the top-level > Distutils __inst__.py file lives), .pydistutils.cfg in the user's > home directory, and setup.cfg in the current directory. [snip] I totally agree with the site-home-currentdir structure, but shouldn't the site level file live someplace apart from the code? If you customize the site file, you probably don't want your changes clobbered by a subsequent distutils install. sys.prefix + '/etc', perhaps? ============================================================================= michaelMuller = mmuller(a)enduden.com | http://www.cloud9.net/~proteus ----------------------------------------------------------------------------- Mantra for the 60's: Tune in, turn on, drop out Mantra for the 90's: Turn on, jack in, jerk off =============================================================================

5 5

Latest bdist_rpm patch
by Harry Henry Gebel 09 May '00

09 May '00

Here is the latest bdist_rpm patch. It gives build_ext the ability to recognize and use the CFLAGS environment variable, it then sets CFLAGS to RPM_OPT_FLAGS in the spec file. It adds an option '--no-rpm-opt-flags' which will cause a spec file to be generated that does not set CFLAGS, this option is turned on automatically for pure python distributions. -- Harry Henry Gebel, Senior Developer, Landon House SBS West Dover Hundred, Delaware PyNcurses ncurses binding for Python http://pyncurses.sourceforge.net

1 0

bdist_rpm and --arch
by Harry Henry Gebel 09 May '00

09 May '00

The response from the rpm list seems to indicate that the problem I am having is an rpm bug in versions < 3.04, and the response on how to get around the bug was "upgrade to 3.04". Rather than require that rpm be a certain version, I am going to remove this option. Anyone who wishes to build under a different architecture than the default can still make a source RPM with Distutils and then run `rpm -bb` from the command line. I am currently adding CFLAGS support so that the i686 and i586 RPMs generated by default on Pentium-class machines will really be i686 and i586 RPMs -- Harry Henry Gebel, Senior Developer, Landon House SBS West Dover Hundred, Delaware PyNcurses ncurses binding for Python http://pyncurses.sourceforge.ne

1 0

More questions on distutils...
by Lyle Johnson 08 May '00

08 May '00

Greg, While working out the stuff for supporting Borland's C/C++ compiler under the Distutils, I've gotten a little confused about the division of responsbilities for the various command and compiler classes. The specific case I'm studying right now is the "built_ext" command (implemented in distutils/command/built_ext.py). If you take a look at the build_extensions() function defined in this class, it appears that he "knows" some things about the "msvc" compiler. It seems like you'd want to avoid letting the various commands know these kinds of details about specific implementations of the CCompiler class. Ideally, they would stick to the basic CCompiler interface, no? I'm not attacking the current implementation -- I know that the primary goal to this point is to get things working. But now I'm faced with the decision about where to put certain Borland-specific flags and so forth -- do they come in solely through the new BCPPCompiler class, or do some of them also filter in through the build_ext command (and his siblings)? Please clue me in ;) I've got the Borland stuff basically working now but I'd like to get it a little cleaner before submitting my proposed patches, etc. Thanks, Lyle

2 1

Re: [Distutils] distutils & Borland's free compiler
by Lyle Johnson 08 May '00

08 May '00

> Excellent! Please do it! I assume you are aware that the starting > point is msvccompiler.py; let me just *strongly* encourage you to > refactor common code out of the MSVCCompiler class and > whatever-you-write (BorlandCCompiler?) rather than submitting a patch > with lots of overlapping code. Maybe I need to double-check the architecture of the distutils with you. It appeared that I (basically) needed to do two things: 1. Create a new class (I was calling it BCPPCompiler, but I'm not too attached to that name ;) which is derived from CCompiler. This class lives in its own module file "bcppcompiler.py" so that it is on the same footing as "msvccompiler.py" (which supports that other company's compiler). 2. Patch distutils/ccompiler.py to add this new compiler into the compiler_class dictionary. 3. Ask Greg about how to let the end-user actually override the default compiler choice for platform "nt". This could be done by passing the keyword argument compiler=bcpp to the new_compiler() factory function, if we had some kind of commmand-line option to let the user do that... Three! Three things! Lyle

2 1

Where to find config files
by Greg Ward 07 May '00

07 May '00

Hi all -- I've finally started coding the Distutils config file stuff, now that I actually have a design of sorts on paper. Anyways, one little procedural issue: how many config files should there be, and where to find them? Here's my initial stab at a policy: def find_config_files (self): """Find as many configuration files as should be processed for this platform, and return a list of filenames in the order in which they should be parsed. The filenames returned are guaranteed to exist and be readable (modulo nasty race conditions). On Unix, there are three possible config files: pydistutils.cfg in the Distutils installation directory (ie. where the top-level Distutils __inst__.py file lives), .pydistutils.cfg in the user's home directory, and setup.cfg in the current directory. On Windows and Mac OS, there are two possible config files: pydistutils.cfg in the Python installation directory (sys.prefix) and setup.cfg in the current directory.""" Code forthcoming... Greg -- Greg Ward - Unix bigot gward(a)python.net http://starship.python.net/~gward/ Are we THERE yet?

1 0

Update bdist_rpm patch
by Harry Henry Gebel 06 May '00

06 May '00

Okay, here is a new version of the bdist_rpm patch, it adds the following options: --spec-only - This option just generates a spec file and then stops --tar-only - This option just generates a tarball and stops. The command `rpm -ta program_name.tar.gz` will then build an RPM --no-remove - bdist_rpm now by default will remove MANIFEST, redhat/, and program_name.tar.gz when run as root. This option will disable this behavior --arch - This option will build the RPM for the specified architecture. One important consideration for --arch : Unless ./setup.py can be made to recognize the rpm's CFLAGS it will generate the same object code no matter what architecture you specify. In this case --arch will only effect what rpm THINKS the package is. I looked through the ccompiler and unixccompiler modules and didn't find any way to pass any CFLAGS in either through the command line or through an environment variable, am I missing it or has this feature not been added yet? -- Harry Henry Gebel, Senior Developer, Landon House SBS West Dover Hundred, Delaware PyNcurses ncurses binding for Python: http://pyncurses.sourceforge.net

2 4

Distutils-SIG May 2000