[Distutils] [setuptools] include files and eggs
Phillip J. Eby
pje at telecommunity.com
Mon Aug 7 04:49:12 CEST 2006
At 04:23 PM 8/6/2006 -0700, Ronald Oussoren wrote:
>Several python packages, such as Numeric, will install header files
>during installation. It seems to me that the best way to support this
>in a world of eggs is to add those include files to eggs and teach
>setuptools to use those includefiles in build_ext.
>The attached patch does just that.
Thank you for taking the time to do this; even if I don't use the patch,
your analysis of the problem and proof of concept is very helpful. I wish
more people would do this when they propose expansive additions to
> It adds an egg_info.writer that
>writes all distribution.headers to the egg-info directory and patches
>build_ext to add header files from all active distributions to the
>I'm not entirely sure that this patch is correct, although it seems
>to work just fine.
One thing that might be a problem is that you're always effectively writing
the headers in Unix file format, so they will appear that way regardless of
origin and destination platform. I don't know if that's the right thing to
do. Perhaps somebody with some experience could speak to this, as far as
whether any Windows or Mac tools might have a problem with this?
> The egg_info.writer for include files adds a
>directory with files to the meta-data, the documentation and API seem
>to assume that writers will write a single file.
I don't really see it as a problem, I'll take a look at the docs.
> The patch to
>build_ext feels hackish
I presume you mean due to the temporary directory. Probably the right
solution is to allow access to metadata filenames in 0.7, so that the
temporary directory can be managed by the usual extraction techniques for
zipped eggs. I have seen other places where file access for metadata would
be useful, so the addition wouldn't just be for this.
>and I'm not sure if adding all active
>distributions is the right thing to do, maybe only header files from
>distributions that are mentioned as setup_requires should be added.
>That would avoid possible problems during upgrades of packages.
Using the setup_requires requirements is probably the right thing to do.
I won't be able to include this new feature in 0.6, so it'll be part of 0.7
instead. But it seems like a terrific idea, and fits well with setuptools'
design principles. It provides backward compatibility for plain distutils
packages that provide headers, and it lets setuptools users depend on those
headers. Thank you very much!
Of course, since it is such a useful and sensible idea, it will of course
be incompatible with Debian Python policy.
Sadly, that isn't a joke, as I'm sure there will be screams of bloody
murder regarding the idea of putting headers inside directories inside
Python version-specific library directories, not the least since they will
want to split such packages into a plain and "devel" version.
There will then be much wailing and gnashing of teeth by people trying to
develop SVN-based packages atop Debian, since setuptools will notice the
presence of the Debian-installed packages and try to use their (missing)
header files when the user attempts to build something.
The Debian team will see nothing wrong with this because obviously the user
should've known by osmosis that a -devel package was needed, and... [sigh]
Okay, no point in continuing that rant. Instead, a counter-meme is
required, to permit easy education of developers. Maybe something like,
"Debian thinks that users aren't developers, so they strip out everything
that might be used to develop software. Make sure you have a '-devel'
package for *everything* on your system, because otherwise it's impossible
to tell what things will be broken when you try to develop software." That
has at least some chance of allowing things to work, at the cost of having
to be repeated to every single developer that encounters the problem, and
for every Python package that might be their point of entry to discovering
Unfortunately, this is a religious issue, which means it is not solvable by
merely technical means. If it were possible to have a mapping from PyPI
projects to Debian packages, it would at least be possible in theory for
setuptools to notify the user of the missing Debian package. However,
Debian policy conflates various meanings of "package" in a way that is
utterly incompatible with such a mapping, only manual mapping is
possible. Essentially, there would have to be some kind of *manually
maintained* global mapping of PyPI project names to Debian package
names. If somebody in the Debian community wants to create and maintain
the thing, then in theory at least, setuptools could use it to tell the
user to install the Debian packages, or invoke the corresponding package
Maybe in setuptools 0.7, I should provide some way of providing alternative
installation hooks, and then encourage people who complain about
Debian-related problems to write a plugin that wraps the Debian
tools. Perhaps there might even be some way to use existing Debian package
metadata directories to find out what Debian package(s) need to be
installed in order to mimic a direct installation of a particular PyPI project.
More information about the Distutils-SIG