[Distutils] making Extensions more flexible

[Stefan Seefeld]

hi there,

I'v been having problems with extension compilation and I'm
looking for ways to enhance the situation.

Let me start with an extreme case, where I have an existing
C/C++ library with autotools based build system, and I want
to provide a python wrapper for that. Let's assume that my
library doesn't exist stand-alone, i.e. I want to make it
a python module with minimal changes, i.e. for example only
a single additional file that provides the interface
python <-> C/C++.

That's currently not possible, as the 'build_ext' command
has a very specific idea of how to process source files into
a loadable python module.

How can this be enhanced ? I can of course write my own 'build_ext'
that is a wrapper around my existing build system, i.e. which
calls 'configure', 'make', etc.
One of the problems is that there is still the 'Extension' class
which is used. With my new 'build_ext', I can't just list the
individual src files, as they are opaque to the library I'm wrapping.

A way out of this situation would be to allow arbitrary keywords
to 'setup()', which are stored as attributes in the 'distribution'
object, and then accessible to the commands.
So if I add my own attributes *and* my own commands, they can share
some specific knowledge.

For example

class my_build(Command):
   def run(self):
     my_data = self.distribution.my_data
     ...process them...

setup(cmdclass={'my_build':my_build},
       my_data="some data to be processed by 'my_build'",
       ...)

you get the idea...

This covers the extreme case where I have to write my own commands
since the existing ones aren't suitable. However, there is already
quite a bit that can be done to make the existing 'build_ext' more
flexible.

For example, I have situations where different source files need
different commands to be compiled. Think about the use of 'lex',
'bison', and other tools. This could be achieved in two steps:

* provide an abstract 'Target' class that knows how to process
   its 'input files'
* derive 'Extension' from 'Target', and make it be composable out
   of (sub)targets

That would work almost as traditional Makefiles, such that one could
compose a tree of targets, each node its own build rules. The
variables (CXX, CPPFLAGS, LEX) could still be provided through
the 'build_ext' command.

What do you think ?

Regards,
		Stefan