distutils problem with interdependent modules

[Daniel Wexler]

I have a moderately complex set of C extension modules that
have interdependencies in the C code.  I have this system compiling
and working as separate modules (i.e.. not a package) with my own
makefile system.  It compiles and runs perfectly on both Linux and
Windows.  I am having difficulty in setting up the proper linking
as I convert to a package format using distutils.

Here's a simplified description of the problem:

C1, C2  - two pure C libraries (i.e.. not Python extensions)
P1, P2 - two C extension modules

C2 requires functions in C1
P1 requires functions in C2
P2 requires functions in P1 and C1

In my current, working, makefile (and VC++ project file) system,
when I build the .so for C2, I add a hardcoded path to the C1 DSO
using something like this:

  ld -shared --whole-archive C2.a /usr/local/lib/flarg/C1.so -o C2.so

It sucks having to use a hardcoded path since you cannot really
build and install very well.  This is because on Linux, the
LD_LIBRARY_PATH is only used to find libraries that use the
file name format:  "libfoo.so", or in my case "libC1.so" rather
than just "C1.so".

The same sort of problem happens when I try to link the other
C extension modules, since they also have to specify the pure C
and other C extension modules on the link line.  The naming problem
might not be so bad for my pure C libraries, but it seems strange,
and actually against the distutils naming scheme, to import a Python
C extension using "import libfoo" as opposed to "import foo", right?

Are there any examples of a Python C extension package with
interdependent modules that I can look at?

I have the beginnings of a setup.py file that correctly finds all the files
but has trouble building.  I start by manually compiling the two pure
C libraries using 'os.system("make -C lib/C2")' and then let the setup()
function take over afterwards.  It seems to start building my first python
C extension correctly, but I'm having trouble with the link step.

Specifically, I don't know how I should properly reference the
dependent libraries.  For example, if I use the "extra_objects"
keyword, then I have to put in an extension (contrary to what the
documentation says, which I don't understand) like this:

  extra_objects=["lib/C1/C1.so", "lib/C2/C2.so"]

The documentation indicates I should leave off the .so, but when I do
it doesn't append .so on the link line (I haven't tried it in Windows yet).

Instead, I guess I can rename my modules using the "libC1" prefix,
which then allows me to use the "libraries" and "library_dirs" keywords.
I'm just about to try that out, but I thought I'd check around on the
newsgroups to see if this has been addressed first.

A separate questions is regarding the package_dir option and how
it affects the file names used in the Extension() constructor.  The
documentation lead me to believe that if I had this:

    package_dir = {'parc' : 'lib'},
    packages = ['parc', 'parc.P1', 'parc.P2'],

Then my extension files should live in <root>/lib/{P1,P2}, and I
would expect to use just the local file name in the Extension()
constructor, but instead I find that I have to put the entire path
to the source file like this:

    ext_modules = [Extension("parc.P1", ["lib/P1/foo.c", "lib/P1/bar.c"])]


Thanks for any help,

Daniel Wexler
wex at flarg.com