[Distutils] [setuptools] include files and eggs

Andrew Straw strawman at astraw.com
Mon Aug 7 22:13:34 CEST 2006

Phillip J. Eby wrote:
> Of course, since it is such a useful and sensible idea, it will of course 
> be incompatible with Debian Python policy.
> Sadly, that isn't a joke, as I'm sure there will be screams of bloody 
> murder regarding the idea of putting headers inside directories inside 
> Python version-specific library directories, not the least since they will 
> want to split such packages into a plain and "devel" version.
Hmm, AFAICT they haven't screamed bloody murder regarding numpy, which
has been including its header files in an egg-compatible way for a long
time (since its inception almost a year ago?). Now that it's in beta,
it's surely getting some widespread use, presumably even by Debian
Developers. (To be sure, some people have screamed bloody murder about
this, but they haven't been DDs, to my knowledge.)
> There will then be much wailing and gnashing of teeth by people trying to 
> develop SVN-based packages atop Debian, since setuptools will notice the 
> presence of the Debian-installed packages and try to use their (missing) 
> header files when the user attempts to build something.
> The Debian team will see nothing wrong with this because obviously the user 
> should've known by osmosis that a -devel package was needed, and...  [sigh]
> Okay, no point in continuing that rant.  Instead, a counter-meme is 
> required, to permit easy education of developers.  Maybe something like, 
> "Debian thinks that users aren't developers, so they strip out everything 
> that might be used to develop software.  Make sure you have a '-devel' 
> package for *everything* on your system, because otherwise it's impossible 
> to tell what things will be broken when you try to develop software."  That 
> has at least some chance of allowing things to work, at the cost of having 
> to be repeated to every single developer that encounters the problem, and 
> for every Python package that might be their point of entry to discovering 
> the problem.
Well, I'm just glad that my Debian systems (presumably) aren't cluttered
up with the cruft for developing perl programs, just the bare minimum
necessary for them to run, thank you very much. :) Those perl developers
probably feel the same about Python.

Seriously, there are reasons for Debian's seeming madness. I certainly
don't understand them all, but they include: being able to install
working software with minimal footprint, modularly being able to add
documentation and development abilities, minimizing server space
required to house a Python package distribution that might have binary
extension modules that run on all the architectures that Debian
supports. In other words, Debian is a lot of things to a lot of people,
and some have resulted in decisions that are different from those that
would have been made if Debian was designed to prioritize Python
user/developer experiences over other considerations.

> Unfortunately, this is a religious issue, which means it is not solvable by 
> merely technical means.  If it were possible to have a mapping from PyPI 
> projects to Debian packages, it would at least be possible in theory for 
> setuptools to notify the user of the missing Debian package.  However, 
> Debian policy conflates various meanings of "package" in a way that is 
> utterly incompatible with such a mapping, only manual mapping is 
> possible.  Essentially, there would have to be some kind of *manually 
> maintained* global mapping of PyPI project names to Debian package 
> names.  If somebody in the Debian community wants to create and maintain 
> the thing, then in theory at least, setuptools could use it to tell the 
> user to install the Debian packages, or invoke the corresponding package 
> installation tools.
I have the beginnings of a related, but not identical, mapping at
http://stdeb.python-hosting.com/wiki/stdeb_all.cfg This is not exactly
what you describe above, but it's a mapping for use by stdeb
(setuptools-to-debian-source-package converter) that goes from PyPI
names (well, the "name" field of calls to setuptools.setup() -- is that
the PyPI name?) to the names of packages created by stdeb and other
Debian packages required or conflicting.

I welcome discussions of how something like this could be expanded to be
used by easy_install so that it could query the debian dpkg system to
ask if a requirement could be installed based on the Debian repository.
It may not fit into the scope of stdeb very easily, but if it does
there'd be at least one user and maintainer -- me.

stdeb is currently completely non-Debian Python policy compliant. Among
other things, it bundles a Python package distribution in a single .deb,
which, as you indicate, is not the Debian Way. So, this means that my
mapping is basically one-to-one, whereas a one-to-many mapping is needed
to deal with Debian's official packages splitting a single Python
package distribution into multiple .debs. For use by setuptools to
automatically find and download packages, I can see we'd have to tackle
this issue, at least. Which might also move stdeb towards producing
Debian-policy compliant packages.
> Maybe in setuptools 0.7, I should provide some way of providing alternative 
> installation hooks, and then encourage people who complain about 
> Debian-related problems to write a plugin that wraps the Debian 
> tools.  Perhaps there might even be some way to use existing Debian package 
> metadata directories to find out what Debian package(s) need to be 
> installed in order to mimic a direct installation of a particular PyPI project.
The closest thing I can think of is looking at the source package and
installing all the "binary" packages that it is capable of producing
that work on your architecture for the version of Python you're
concerned with.

Another issue is making pkg_resources.require() aware of Debian such
that it can query Debian-installed packages that happen not to be built
with setuptools. Hopefully these will be fewer over time, but currently
there are rather few built with setuptools. Or perhaps you see the
mapping as achieving this?

More information about the Distutils-SIG mailing list