making Extensions more flexible

hi there, I'v been having problems with extension compilation and I'm looking for ways to enhance the situation. Let me start with an extreme case, where I have an existing C/C++ library with autotools based build system, and I want to provide a python wrapper for that. Let's assume that my library doesn't exist stand-alone, i.e. I want to make it a python module with minimal changes, i.e. for example only a single additional file that provides the interface python <-> C/C++. That's currently not possible, as the 'build_ext' command has a very specific idea of how to process source files into a loadable python module. How can this be enhanced ? I can of course write my own 'build_ext' that is a wrapper around my existing build system, i.e. which calls 'configure', 'make', etc. One of the problems is that there is still the 'Extension' class which is used. With my new 'build_ext', I can't just list the individual src files, as they are opaque to the library I'm wrapping. A way out of this situation would be to allow arbitrary keywords to 'setup()', which are stored as attributes in the 'distribution' object, and then accessible to the commands. So if I add my own attributes *and* my own commands, they can share some specific knowledge. For example class my_build(Command): def run(self): my_data = self.distribution.my_data ...process them... setup(cmdclass={'my_build':my_build}, my_data="some data to be processed by 'my_build'", ...) you get the idea... This covers the extreme case where I have to write my own commands since the existing ones aren't suitable. However, there is already quite a bit that can be done to make the existing 'build_ext' more flexible. For example, I have situations where different source files need different commands to be compiled. Think about the use of 'lex', 'bison', and other tools. This could be achieved in two steps: * provide an abstract 'Target' class that knows how to process its 'input files' * derive 'Extension' from 'Target', and make it be composable out of (sub)targets That would work almost as traditional Makefiles, such that one could compose a tree of targets, each node its own build rules. The variables (CXX, CPPFLAGS, LEX) could still be provided through the 'build_ext' command. What do you think ? Regards, Stefan

Stefan Seefeld wrote:
hi there,
I'v been having problems with extension compilation and I'm looking for ways to enhance the situation.
Let me start with an extreme case, where I have an existing C/C++ library with autotools based build system, and I want to provide a python wrapper for that. Let's assume that my library doesn't exist stand-alone, i.e. I want to make it a python module with minimal changes, i.e. for example only a single additional file that provides the interface python <-> C/C++.
That's currently not possible, as the 'build_ext' command has a very specific idea of how to process source files into a loadable python module.
How can this be enhanced ? I can of course write my own 'build_ext' that is a wrapper around my existing build system, i.e. which calls 'configure', 'make', etc.
Right, that's how you'd do it. If you come up with a good implementation, I'd suggest you post it as patch to SF. I can't see why you would want build on build_ext for this, though. Writing a new command from scratch using the available tools would be much cleaner. The code for mx_build_unixlib you find in the mxSetup.py module of egenix-mx-experimental is an example of how this can be done. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Aug 28 2003)
Python/Zope Products & Consulting ... http://www.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2003-08-12: Released eGenix mx Extensions for Python 2.3 ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

M.-A. Lemburg wrote:
How can this be enhanced ? I can of course write my own 'build_ext' that is a wrapper around my existing build system, i.e. which calls 'configure', 'make', etc.
Right, that's how you'd do it. If you come up with a good implementation, I'd suggest you post it as patch to SF.
I can't see why you would want build on build_ext for this, though. Writing a new command from scratch using the available tools would be much cleaner.
agreed. Though the issue I see with this is that some of the other commands which I'd still use rely directly or indirectly on build_ext. For example, the existence of Extension objects in the distribution object is used to decide where to compile ('pure' or not), and the automatic file list that is generated for commands such as 'sdist' will inspect the Extensions, too. So the question is how I can provide my own extension mechanism in a transparent way, i.e. such that the rest of the framework will be able to deduce all it has to know to work.
The code for mx_build_unixlib you find in the mxSetup.py module of egenix-mx-experimental is an example of how this can be done.
yeah, I had a quick look, before I wrote my own. Again, the problem I see is not so much with the command itself as with the way to integrate that. Regards, Stefan
participants (2)
-
M.-A. Lemburg
-
Stefan Seefeld