Hello all!
I've been playing around with Distutils-0.8.2 and just have a few
questions. Before I start though, thanks for all your work on this so
far, it is *very* nice and much needed!
I just have two questions:
1. I'm working on writing a setup.py for an extension that uses gnome
and glib libraries, and was curious if there was a general way anyone
has already thought of to handle getting information from gnome-config
and glib-config calls into a form that you can input into Distutils.
For instance, if I do something like the following:
>>> import commands
>>> gthread_cflags = commands.getoutput("glib-config --cflags gthread")
>>> print gthread_cflags
-I/usr/local/include/glib12 -D_THREAD_SAFE
I can get the options you would pass. It would be useful to have
something to process this like:
def parse_cflag_config(configure_script, library)
which would then return two lists, the include dirs and the macros. So
they you could do something in your setup.py like:
gthread_includes, gthread_macros = parse_cflag_config("glib-config",
"gthread")
Of course, this would only work on UNIX, but that is where the config
calls are anyways :-)
Is this something that would be generally useful to anyone? Does this
approach seem like a good idea? Is there already a better plan in
place to handle this?
2. Going along with this first problem, I was also wondering how to
pass specific compiler flags. The issue I have is that on FreeBSD,
there is a FreeBSD specific option for linking the thread libraries:
'-pthread' (what this option does is link in libc_r and not libc, if
anyone cares. I'm afraid I'm not smart enough to more about it than
that :-).
Anyways, is there a way to pass a flag like '-pthread'? Apologies
if I missed it!
Thanks much for listening and thanks again for the wonderful tool.
Brad
Hi,
I'm currently trying to make up my mind how Distutils and building Debian
packages fit together.
I have to impression we're faced with something like a chicken and egg
problem. ...
Since not everybody knows the Debian package format, I'll give a short
introduction:
A short story of .deb's and Debian packages
-------------------------------------------
A .deb package is in fact really, really simple:
freefly;158> ar tv python-ldap_1.8-1_i386.deb
rw-r--r-- 0/0 4 Nov 30 15:58 1999 debian-binary
rw-r--r-- 0/0 1078 Nov 30 15:58 1999 control.tar.gz
rw-r--r-- 0/0 52562 Nov 30 15:58 1999 data.tar.gz
freefly;159> ar x python-ldap_1.8-1_i386.deb
freefly;160> cat debian-binary
2.0
freefly;161> tar tvpzf control.tar.gz
drwxr-xr-x root/root 0 1999-11-30 15:58:41 ./
-rwxr-xr-x root/root 553 1999-11-30 15:58:05 ./postinst
-rwxr-xr-x root/root 461 1999-11-30 15:58:05 ./prerm
-rw-r--r-- root/root 660 1999-11-30 15:58:41 ./md5sums
-rw-r--r-- root/root 392 1999-11-30 15:58:41 ./control
freefly;162> tar tvpzf data.tar.gz
drwxr-xr-x root/root 0 1999-11-30 15:58:04 ./
drwxr-xr-x root/root 0 1999-11-30 15:58:01 ./usr/
drwxr-xr-x root/root 0 1999-11-30 15:58:01 ./usr/lib/
drwxr-xr-x root/root 0 1999-11-30 15:58:01 ./usr/lib/python1.5/
...
-rw-r--r-- root/root 228 1999-11-30 15:48:07 ./usr/share/doc-base/python-ldap
The nice thing is that you can simply unpack the file with standard Unix
tools. The same is true for the reverse: You can build a .deb with ar and
tar only.
What's the stuff in control.tar.gz ?
postinst and prerm (and preinst and postrm) are optional scripts that are
called at various points of the installation/deinstallation of a package;
control is a text file describing the package:
freefly;163> cat control
Package: python-ldap
Version: 1.8-1
Section: net
Priority: optional
Architecture: i386
Depends: libc6 (>= 2.1), libopenldap1
Installed-Size: 99
Maintainer: Gregor Hoffleit <flight(a)debian.org>
Description: An LDAP module for Python.
This module provides an Python interface to the LDAP client library
(LDAP is the Lightweight Directory Access Protocol). It has been
compiled with OpenLDAP.
Building a valid .deb package is as simple as
echo 2.0 >debian-binary
vi control
tar -c -z -f control.tar.gz ./control
tar -c -z -f data.tar.gz -C build .
ar cv my1stdeb-0.0.deb debian-binary control.tar.gz data.tar.gz
Now while this is a valid .deb file, there's more to a valid Debian package
than this: Packages have to follow some rules in order to
(a) not to hose the system were they're unpacked
(b) fit into the Debian archive scheme
(c) support building from source
(d) support auto-building from source
These rules are put together in the Debian Policy manual
(http://www.debian.org/doc/debian-policy/) and the Debian Packaging manual.
The most interesting things in our context are certainly (c) and (d).
A valid Debian package should be built from a source package. A source
package consists of a tarball of the upstream version (xyz_1.2.orig.tar.gz),
a diff file with Debian's changes to the package (xyz_1.2-5.diff.gz) and a
text file with a description of the package (xyz_1.2-5.dsc) including things
like md5sums of the source files and GPG signed by the maintainer of the
package. (If you have these three files, you can unpack the source tree with
"dpkg-source -x xyz_1.2-5.dsc", or with
tar -xzpf xyz_1.2.orig.tar.gz
zcat xyz_1.2-5.diff.gz | patch -p1
Debian's equivalent to an RPM spec file is a ./debian directory in the
source tree. Typically, this directory looks like this:
freefly;193> cd python-ldap-1.8/
freefly;194> ls -ld debian/*
-rw-r--r-- 1 flight flight 323 Mai 10 21:28 debian/README.Debian
-rw-r--r-- 1 flight flight 203 Mai 10 21:28 debian/TODO.Debian
-rw-r--r-- 1 flight flight 1644 Mai 10 21:28 debian/changelog
-rw-r--r-- 1 flight flight 392 Mai 10 21:28 debian/control
-rw-r--r-- 1 flight flight 799 Mai 10 21:28 debian/copyright
-rw-r--r-- 1 flight flight 228 Mai 10 21:28 debian/doc-base
-rw-r--r-- 1 flight flight 133 Mai 10 21:28 debian/postinst
-rw-r--r-- 1 flight flight 130 Mai 10 21:28 debian/prerm
-rwxr-xr-x 1 flight flight 1576 Mai 10 21:28 debian/rules*
The most important thing is debian/rules (note the connotation ;-), which is
in fact an (executable) Makefile with targets "build", "clean",
"binary-indep", "binary-arch" and "binary".
Provided the tools used in this rules Makefile are installed, building all
packages from a source tree is done by calling
debian/rules binary
"debian/rules binary-indep" would build only the packages that are flagged
architecture-independent (like a 100% pure Python package); "debian/rules
binary-arch" would build the architecture-dependent packages for the
architecture of the system we're running on (e.g. i386, m68k, sparc,
hurd-i386 or perhaps bsd-i386 or cygwin-i386).
For a normal package, "rules build" calls "./configure; make", "rules clean"
calls "make clean", "rules binary" calls "make DESTDIR='pwd'/debian/tmp
install" and "dh_builddeb" (from a simplified view).
Needless to say that Debian's build tools (and autobuild daemon's) rely on
the presence of ./debian/rules to build binary packages for every
architecture from their source packages.
Enters Distutils
----------------
So where exactly does Distutils fit into this picture, how much sense makes
bdist_deb, and what should it do ?
From the theory above, you see that it's possible and even quite easy to
write a bdist_deb target that builds a valid .deb file (using ar and tar)
for any Distutil'ed package. The problem is that that's still only a
third-class Debian package: Fine for rapid-prototyping of .deb packages, but
without support for Debian's infrastructure.
If we'd like to do better, we have to support debian/rules somehow. I'm
still not sure what's the best way to do this, and how bdist's "source
distributions" fit into this.
Possible ideas:
bdist_deb could create a source archive with a debian/rules skeleton that
just works(tm). The debian/rules skeleton would just call up setup.py in
different places. I.e. bdist_deb source distribution command creates a
source package; this source package uses (in debian/rules) simple setup.py
targets like "clean", "build" and "install" and finally builds the .deb file
using standard Debian tools.
In an ideal world (any Distutil'ed package maps perfectly in a Debian
package) this would be fine. In our world, Debian maintainers still would
have to merge their changes with new upstream versions, therefore providing
a debian/rules skeleton is perhaps not enough.
Maybe the bdist_deb binary distribution command then could simply call up
"debian/rules binary" to build the binary packages.
A little bit more elaborated thoughts to come, I hope.
Gregor
Here is a new bdist_rpm patch that adds the following variables to
package_data:
Strings:
icon - path (relative to setup.py) to an icon which will be used for the
package in graphical rpm front-ends
distribution_name - placed in spec file 'Distribution' field
pre, post, preun, postun, prep, build, install, clean - spec file scripts
(defaults are still provided for prep, build, clean, and install that
should work 99% of the time)
String or number:
serial - placed in spec file 'Serial' field
Lists of strings or strings:
provides, requires, build_requires, conflicts, obsoletes - placed in
appropriate fields in spec file
The default value of 'group' in package_data has been changed from
'Applications' to 'Development/Libraries'. Type checking has been cleaned
up a little. The default addition of 'README' and 'README.txt' to the doc
files can be overridden by defining doc as a string rather than as a list
of strings. Defining doc as '' will prevent any files including 'README'
and 'README.txt' from being include in the doc files.
I modified clean.py so that 'clean -a' removes the 'build/rpm'
directory. (The rpm program handles cleaning of the 'build/rpm/BUILD'
directory, unless 'bdist_rpm --no-clean' is specified. (Or should
--no-clean be the default for bdist_rpm?))
--
Harry Henry Gebel, Senior Developer, Landon House SBS
West Dover Hundred, Delaware
PyNcurses ncurses binding for Python http://pyncurses.sourceforge.net
Greg Ward wrote:
[snip]
> On Unix, there are three possible config files: pydistutils.cfg in
> the Distutils installation directory (ie. where the top-level
> Distutils __inst__.py file lives), .pydistutils.cfg in the user's
> home directory, and setup.cfg in the current directory.
[snip]
I totally agree with the site-home-currentdir structure, but shouldn't the
site level file live someplace apart from the code? If you customize the
site file, you probably don't want your changes clobbered by a subsequent
distutils install. sys.prefix + '/etc', perhaps?
=============================================================================
michaelMuller = mmuller(a)enduden.com | http://www.cloud9.net/~proteus
-----------------------------------------------------------------------------
Mantra for the 60's: Tune in, turn on, drop out
Mantra for the 90's: Turn on, jack in, jerk off
=============================================================================
Here is the latest bdist_rpm patch.
It gives build_ext the ability to recognize and use the CFLAGS environment
variable, it then sets CFLAGS to RPM_OPT_FLAGS in the spec file. It adds
an option '--no-rpm-opt-flags' which will cause a spec file to be
generated that does not set CFLAGS, this option is turned on automatically
for pure python distributions.
--
Harry Henry Gebel, Senior Developer, Landon House SBS
West Dover Hundred, Delaware
PyNcurses ncurses binding for Python http://pyncurses.sourceforge.net
The response from the rpm list seems to indicate that the problem I am
having is an rpm bug in versions < 3.04, and the response on how to get
around the bug was "upgrade to 3.04". Rather than require that rpm be a
certain version, I am going to remove this option. Anyone who wishes to
build under a different architecture than the default can still make a
source RPM with Distutils and then run `rpm -bb` from the command line.
I am currently adding CFLAGS support so that the i686 and i586 RPMs
generated by default on Pentium-class machines will really be i686 and
i586 RPMs
--
Harry Henry Gebel, Senior Developer, Landon House SBS
West Dover Hundred, Delaware
PyNcurses ncurses binding for Python http://pyncurses.sourceforge.ne
Greg,
While working out the stuff for supporting Borland's C/C++ compiler under
the Distutils, I've gotten a little confused about the division of
responsbilities for the various command and compiler classes. The specific
case I'm studying right now is the "built_ext" command (implemented in
distutils/command/built_ext.py).
If you take a look at the build_extensions() function defined in this class,
it appears that he "knows" some things about the "msvc" compiler. It seems
like you'd want to avoid letting the various commands know these kinds of
details about specific implementations of the CCompiler class. Ideally, they
would stick to the basic CCompiler interface, no?
I'm not attacking the current implementation -- I know that the primary goal
to this point is to get things working. But now I'm faced with the decision
about where to put certain Borland-specific flags and so forth -- do they
come in solely through the new BCPPCompiler class, or do some of them also
filter in through the build_ext command (and his siblings)?
Please clue me in ;) I've got the Borland stuff basically working now but
I'd like to get it a little cleaner before submitting my proposed patches,
etc.
Thanks,
Lyle
> Excellent! Please do it! I assume you are aware that the starting
> point is msvccompiler.py; let me just *strongly* encourage you to
> refactor common code out of the MSVCCompiler class and
> whatever-you-write (BorlandCCompiler?) rather than submitting a patch
> with lots of overlapping code.
Maybe I need to double-check the architecture of the distutils with you. It
appeared that I (basically) needed to do two things:
1. Create a new class (I was calling it BCPPCompiler, but I'm not too
attached to that name ;) which is derived from CCompiler. This class
lives in its own module file "bcppcompiler.py" so that it is on the
same footing as "msvccompiler.py" (which supports that other company's
compiler).
2. Patch distutils/ccompiler.py to add this new compiler into the
compiler_class dictionary.
3. Ask Greg about how to let the end-user actually override the default
compiler choice for platform "nt". This could be done by passing the
keyword argument compiler=bcpp to the new_compiler() factory function,
if we had some kind of commmand-line option to let the user do that...
Three! Three things!
Lyle
Hi all --
I've finally started coding the Distutils config file stuff, now that I
actually have a design of sorts on paper. Anyways, one little
procedural issue: how many config files should there be, and where to
find them? Here's my initial stab at a policy:
def find_config_files (self):
"""Find as many configuration files as should be processed for this
platform, and return a list of filenames in the order in which they
should be parsed. The filenames returned are guaranteed to exist
and be readable (modulo nasty race conditions).
On Unix, there are three possible config files: pydistutils.cfg in
the Distutils installation directory (ie. where the top-level
Distutils __inst__.py file lives), .pydistutils.cfg in the user's
home directory, and setup.cfg in the current directory.
On Windows and Mac OS, there are two possible config files:
pydistutils.cfg in the Python installation directory (sys.prefix)
and setup.cfg in the current directory."""
Code forthcoming...
Greg
--
Greg Ward - Unix bigot gward(a)python.net
http://starship.python.net/~gward/
Are we THERE yet?
Okay, here is a new version of the bdist_rpm patch, it adds the following
options:
--spec-only - This option just generates a spec file and then stops
--tar-only - This option just generates a tarball and stops. The command
`rpm -ta program_name.tar.gz` will then build an RPM
--no-remove - bdist_rpm now by default will remove MANIFEST, redhat/, and
program_name.tar.gz when run as root. This option will
disable this behavior
--arch - This option will build the RPM for the specified
architecture.
One important consideration for --arch : Unless ./setup.py can be made to
recognize the rpm's CFLAGS it will generate the same object code no matter
what architecture you specify. In this case --arch will only effect what
rpm THINKS the package is. I looked through the ccompiler and
unixccompiler modules and didn't find any way to pass any CFLAGS in either
through the command line or through an environment variable, am I missing
it or has this feature not been added yet?
--
Harry Henry Gebel, Senior Developer, Landon House SBS
West Dover Hundred, Delaware
PyNcurses ncurses binding for Python: http://pyncurses.sourceforge.net
