[Distutils] [setuptools] include files and eggs

[Ronald Oussoren]

On 6-aug-2006, at 19:49, Phillip J. Eby wrote:

> At 04:23 PM 8/6/2006 -0700, Ronald Oussoren wrote:
>> Hi,
>>
>> Several python packages, such as Numeric, will install header files
>> during installation. It seems to me that the best way to support this
>> in a world of eggs is to add those include files to eggs and teach
>> setuptools to use those includefiles in build_ext.
>>
>> The attached patch does just that.
>
> Thank you for taking the time to do this; even if I don't use the  
> patch, your analysis of the problem and proof of concept is very  
> helpful.  I wish more people would do this when they propose  
> expansive additions to setuptools.  :)

I had to do something on the plane ;-).
>
>
>>  It adds an egg_info.writer that
>> writes all distribution.headers to the egg-info directory and patches
>> build_ext to add header files from all active distributions to the
>> include path.
>>
>> I'm not entirely sure that this patch is correct, although it seems
>> to work just fine.
>
> One thing that might be a problem is that you're always effectively  
> writing the headers in Unix file format, so they will appear that  
> way regardless of origin and destination platform.  I don't know if  
> that's the right thing to do.  Perhaps somebody with some  
> experience could speak to this, as far as whether any Windows or  
> Mac tools might have a problem with this?

Mac tools shouldn't be a problem here, current versions of MacOS are  
unix and use unix EOL conventions. I don't know how windows compilers  
deal with unix-style files.

BTW. I don't understand why you think the headers are written in Unix  
format as my patch opens files in text mode. Meta data is read as  
binary data and then writting out in text mode. That could cause  
problems: if the egg contains a header in windows format the  
resulting file will be mangled because the data returned by  
metadata_read contains '\r\n' which write will then happily change  
into '\r\r\n'. This could change the meaning of header file.
>
>>  The patch to
>> build_ext feels hackish
>
> I presume you mean due to the temporary directory.  Probably the  
> right solution is to allow access to metadata filenames in 0.7, so  
> that the temporary directory can be managed by the usual extraction  
> techniques for zipped eggs.  I have seen other places where file  
> access for metadata would be useful, so the addition wouldn't just  
> be for this.

I did mean the temporary directory. On the other hand I don't really  
like extraction stuff for zipped eggs, so I shouldn't complain  
here ;-). I'll write another message on that subject, but basically I  
don't like an ever-growing .python-eggs directory in my home directory.

>
>
>> and I'm not sure if adding all active
>> distributions is the right thing to do, maybe only header files from
>> distributions that are mentioned as setup_requires should be added.
>>
>> That would avoid possible problems during upgrades of packages.
>
> Using the setup_requires requirements is probably the right thing  
> to do.
>
> I won't be able to include this new feature in 0.6, so it'll be  
> part of 0.7 instead.  But it seems like a terrific idea, and fits  
> well with setuptools' design principles.  It provides backward  
> compatibility for plain distutils packages that provide headers,  
> and it lets setuptools users depend on those headers.  Thank you  
> very much!

Adding it to 0.7 is fine. I'm slowly moving my stuff to python 2.5  
and am therefore using the setuptools trunk anyway :-)

>
> Of course, since it is such a useful and sensible idea, it will of  
> course be incompatible with Debian Python policy.

This one is worse, it is incompatible with the packaging policy of  
every Linux distribution, header files must be in -devel packages  
obviously :-(. Maybe --single-version-externally-managed should just  
revert to the current "let's install the header in the python headers  
directory" behaviour, that way you won't get the wrath of the Debian  
packagers while "real" eggs behave as they should.

>
> Sadly, that isn't a joke, as I'm sure there will be screams of  
> bloody murder regarding the idea of putting headers inside  
> directories inside Python version-specific library directories, not  
> the least since they will want to split such packages into a plain  
> and "devel" version.

I know, this can be very annoying and seems to be one of the most  
asked questions on this list (or rather, "how do I install distutils  
manually?").

>
> There will then be much wailing and gnashing of teeth by people  
> trying to develop SVN-based packages atop Debian, since setuptools  
> will notice the presence of the Debian-installed packages and try  
> to use their (missing) header files when the user attempts to build  
> something.
>
> The Debian team will see nothing wrong with this because obviously  
> the user should've known by osmosis that a -devel package was  
> needed, and...  [sigh]

The end result of which may will be that a meme will spread that says  
you shouldn't use the distributors build of python because that is  
broken. I don't envy you here, at least some people that try to build  
their own eggs will see this failure and think it's a bug in setuptools.

>
> Okay, no point in continuing that rant.  Instead, a counter-meme is  
> required, to permit easy education of developers.  Maybe something  
> like, "Debian thinks that users aren't developers, so they strip  
> out everything that might be used to develop software.  Make sure  
> you have a '-devel' package for *everything* on your system,  
> because otherwise it's impossible to tell what things will be  
> broken when you try to develop software."  That has at least some  
> chance of allowing things to work, at the cost of having to be  
> repeated to every single developer that encounters the problem, and  
> for every Python package that might be their point of entry to  
> discovering the problem.
>
> Unfortunately, this is a religious issue, which means it is not  
> solvable by merely technical means.

I'm afraid you're right. What really annoyed me the last time debian  
vs. eggs was discussed here was that some Debian packagers honestly  
believe that they know better than the authors of packages what the  
dependencies of those packages are.

> If it were possible to have a mapping from PyPI projects to Debian  
> packages, it would at least be possible in theory for setuptools to  
> notify the user of the missing Debian package.  However, Debian  
> policy conflates various meanings of "package" in a way that is  
> utterly incompatible with such a mapping, only manual mapping is  
> possible.  Essentially, there would have to be some kind of  
> *manually maintained* global mapping of PyPI project names to  
> Debian package names.  If somebody in the Debian community wants to  
> create and maintain the thing, then in theory at least, setuptools  
> could use it to tell the user to install the Debian packages, or  
> invoke the corresponding package installation tools.

Does debian really split some projects into multiple debian packages  
(excluding -devel variants)? I know they do this for some non-python  
stuff, but haven't noticed this for python software yet. Not that I'm  
likely to notice, the only Debian machine I manage at the moment  
isn't running much python software.

Given that at least some Debian developers think setuptools is  
superfluous I wouldn't hold my breath waiting for a volunteer that  
maintains a mapping from PyPI to debian package names.

Ronald
>