[Distutils] [setuptools] include files and eggs

[Phillip J. Eby]

At 04:23 PM 8/6/2006 -0700, Ronald Oussoren wrote:
>Hi,
>
>Several python packages, such as Numeric, will install header files
>during installation. It seems to me that the best way to support this
>in a world of eggs is to add those include files to eggs and teach
>setuptools to use those includefiles in build_ext.
>
>The attached patch does just that.

Thank you for taking the time to do this; even if I don't use the patch, 
your analysis of the problem and proof of concept is very helpful.  I wish 
more people would do this when they propose expansive additions to 
setuptools.  :)


>  It adds an egg_info.writer that
>writes all distribution.headers to the egg-info directory and patches
>build_ext to add header files from all active distributions to the
>include path.
>
>I'm not entirely sure that this patch is correct, although it seems
>to work just fine.

One thing that might be a problem is that you're always effectively writing 
the headers in Unix file format, so they will appear that way regardless of 
origin and destination platform.  I don't know if that's the right thing to 
do.  Perhaps somebody with some experience could speak to this, as far as 
whether any Windows or Mac tools might have a problem with this?


>  The egg_info.writer for include files adds a
>directory with files to the meta-data, the documentation and API seem
>to assume that writers will write a single file.

I don't really see it as a problem, I'll take a look at the docs.


>  The patch to
>build_ext feels hackish

I presume you mean due to the temporary directory.  Probably the right 
solution is to allow access to metadata filenames in 0.7, so that the 
temporary directory can be managed by the usual extraction techniques for 
zipped eggs.  I have seen other places where file access for metadata would 
be useful, so the addition wouldn't just be for this.


>and I'm not sure if adding all active
>distributions is the right thing to do, maybe only header files from
>distributions that are mentioned as setup_requires should be added.
>
>That would avoid possible problems during upgrades of packages.

Using the setup_requires requirements is probably the right thing to do.

I won't be able to include this new feature in 0.6, so it'll be part of 0.7 
instead.  But it seems like a terrific idea, and fits well with setuptools' 
design principles.  It provides backward compatibility for plain distutils 
packages that provide headers, and it lets setuptools users depend on those 
headers.  Thank you very much!

Of course, since it is such a useful and sensible idea, it will of course 
be incompatible with Debian Python policy.

Sadly, that isn't a joke, as I'm sure there will be screams of bloody 
murder regarding the idea of putting headers inside directories inside 
Python version-specific library directories, not the least since they will 
want to split such packages into a plain and "devel" version.

There will then be much wailing and gnashing of teeth by people trying to 
develop SVN-based packages atop Debian, since setuptools will notice the 
presence of the Debian-installed packages and try to use their (missing) 
header files when the user attempts to build something.

The Debian team will see nothing wrong with this because obviously the user 
should've known by osmosis that a -devel package was needed, and...  [sigh]

Okay, no point in continuing that rant.  Instead, a counter-meme is 
required, to permit easy education of developers.  Maybe something like, 
"Debian thinks that users aren't developers, so they strip out everything 
that might be used to develop software.  Make sure you have a '-devel' 
package for *everything* on your system, because otherwise it's impossible 
to tell what things will be broken when you try to develop software."  That 
has at least some chance of allowing things to work, at the cost of having 
to be repeated to every single developer that encounters the problem, and 
for every Python package that might be their point of entry to discovering 
the problem.

Unfortunately, this is a religious issue, which means it is not solvable by 
merely technical means.  If it were possible to have a mapping from PyPI 
projects to Debian packages, it would at least be possible in theory for 
setuptools to notify the user of the missing Debian package.  However, 
Debian policy conflates various meanings of "package" in a way that is 
utterly incompatible with such a mapping, only manual mapping is 
possible.  Essentially, there would have to be some kind of *manually 
maintained* global mapping of PyPI project names to Debian package 
names.  If somebody in the Debian community wants to create and maintain 
the thing, then in theory at least, setuptools could use it to tell the 
user to install the Debian packages, or invoke the corresponding package 
installation tools.

Maybe in setuptools 0.7, I should provide some way of providing alternative 
installation hooks, and then encourage people who complain about 
Debian-related problems to write a plugin that wraps the Debian 
tools.  Perhaps there might even be some way to use existing Debian package 
metadata directories to find out what Debian package(s) need to be 
installed in order to mimic a direct installation of a particular PyPI project.