linking modules to a shared library
Hi -- I have a problem that is similar to one discussed in a thread from a year ago, http://mail.python.org/pipermail/distutils-sig/2004-September/ 004160.html, but that thread doesn't quite have a resolution in the archives. I'd like to use distutils to build a bunch of python modules that all link to the same shared library. The shared library is all C++ code. On Linux, without distutils, I can build the libraries like this: g++ -o libBase.so -shared base.C # builds the common shared library g++ -o _mymodule.so -shared mymodule.C -lBase # builds a python module # linking to the shared lib. g++ -o _othermodule.so -shared othermodule.C -lBase # and so on _mymodule.so and _othermodule.so can be imported by Python, and share a single copy of the code in libBase.so. On the Mac I can get the same effect with this: g++ -dynamiclib base.C -o libBase.dylib g++ -o _mymodule.so mymodule.C -lBase -bundle -undefined dynamic_lookup -flat_namespace g++ -o _othermodule.so othermodule.C -lBase -bundle -undefined dynamic_lookup -flat_namespace Is it possible to create libBase portably with distutils? It's possible to do it on Linux by subclassing build_ext.build_ext and explicitly using self.compiler.compile() and self.compiler.link_shared_lib() to build the shared library before calling build_ext.build_ext.build_extensions(). But the same thing on Mac OS X only creates libBase.so, whereas I need it to create libBase.dylib. If it matters, I'm using OS X 10.4.2 and gcc 3.3, with Python 2.3.5. Many thanks, Steve -- -- EMail: stephen.langer@nist.gov Phone: (301) 975-5423 -- -- WWW: http://math.nist.gov/mcsd/Staff/SLanger/ Fax: (301) 990-4127 -- -- Mail: NIST; 100 Bureau Drive -- Stop 8910; Gaithersburg, Md 20899-8910 -- -- "I don't think this will work. That's why it's science." -- -- Naomi Langer, 17 Feb 2003 --
On 30-sep-2005, at 19:57, Stephen Langer wrote:
Hi --
I have a problem that is similar to one discussed in a thread from a year ago, http://mail.python.org/pipermail/distutils-sig/2004-September/ 004160.html, but that thread doesn't quite have a resolution in the archives.
I'd like to use distutils to build a bunch of python modules that all link to the same shared library. The shared library is all C++ code. On Linux, without distutils, I can build the libraries like this:
g++ -o libBase.so -shared base.C # builds the common shared library g++ -o _mymodule.so -shared mymodule.C -lBase # builds a python module # linking to the shared lib. g++ -o _othermodule.so -shared othermodule.C -lBase # and so on
_mymodule.so and _othermodule.so can be imported by Python, and share a single copy of the code in libBase.so.
On the Mac I can get the same effect with this:
g++ -dynamiclib base.C -o libBase.dylib g++ -o _mymodule.so mymodule.C -lBase -bundle -undefined dynamic_lookup -flat_namespace g++ -o _othermodule.so othermodule.C -lBase -bundle -undefined dynamic_lookup -flat_namespace
Don't use -flat_namespace. Search the pythonmac SIG archives if you want a more elaborate rant :-).
Is it possible to create libBase portably with distutils? It's possible to do it on Linux by subclassing build_ext.build_ext and explicitly using self.compiler.compile() and self.compiler.link_shared_lib() to build the shared library before calling build_ext.build_ext.build_extensions(). But the same thing on Mac OS X only creates libBase.so, whereas I need it to create libBase.dylib.
I don't know if this is possible, although I'd guess it is not.
If it matters, I'm using OS X 10.4.2 and gcc 3.3, with Python 2.3.5.
Many thanks, Steve
-- -- EMail: stephen.langer@nist.gov Phone: (301) 975-5423 -- -- WWW: http://math.nist.gov/mcsd/Staff/SLanger/ Fax: (301) 990-4127 -- -- Mail: NIST; 100 Bureau Drive -- Stop 8910; Gaithersburg, Md 20899-8910 --
-- "I don't think this will work. That's why it's science." -- -- Naomi Langer, 17 Feb 2003 --
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Oct 2, 2005, at 6:21 AM, Ronald Oussoren wrote:
On 30-sep-2005, at 19:57, Stephen Langer wrote:
g++ -o _othermodule.so othermodule.C -lBase -bundle -undefined dynamic_lookup -flat_namespace
Don't use -flat_namespace. Search the pythonmac SIG archives if you want a more elaborate rant :-).
Ok, I won't. I was just guessing based on what I'd seen elsewhere.
Is it possible to create libBase portably with distutils? It's possible to do it on Linux by subclassing build_ext.build_ext and explicitly using self.compiler.compile() and self.compiler.link_shared_lib() to build the shared library before calling build_ext.build_ext.build_extensions(). But the same thing on Mac OS X only creates libBase.so, whereas I need it to create libBase.dylib.
I don't know if this is possible, although I'd guess it is not.
That's too bad. Is there a reason for it? I'd volunteer to work on modifying distutils so that it can build a .dylib, but I am not an expert on library formats. I don't really know the difference between a .so and a .dylib, except that one of them works and the other doesn't. Can someone point me in the direction of a good reference on the topic? Thanks. -- Steve -- -- EMail: stephen.langer@nist.gov Phone: (301) 975-5423 -- -- WWW: http://math.nist.gov/mcsd/Staff/SLanger/ Fax: (301) 990-4127 -- -- Mail: NIST; 100 Bureau Drive -- Stop 8910; Gaithersburg, Md 20899-8910 -- -- "I don't think this will work. That's why it's science." -- -- Naomi Langer, 17 Feb 2003 --
On 3-okt-2005, at 16:56, Stephen Langer wrote:
I don't know if this is possible, although I'd guess it is not.
That's too bad. Is there a reason for it? I'd volunteer to work on modifying distutils so that it can build a .dylib, but I am not an expert on library formats. I don't really know the difference between a .so and a .dylib, except that one of them works and the other doesn't. Can someone point me in the direction of a good reference on the topic?
I'd guess that nobody has needed this functionality badly enough to volunteer extending distutils :-). The main difference between a .so and a .dylib is that the former is a bundle and the latter is a dylib. Dynamic libraries and loadable objects are two different beasts on OS X. See the manpages for ld(1) and Mach- O(5) for more information. W.r.t. extending distutils: extending the 'clib' might be the best way, although I must say that I've never used that command. Ronald
On Oct 3, 2005, at 1:17 PM, Ronald Oussoren wrote:
I'd guess that nobody has needed this functionality badly enough to volunteer extending distutils :-).
Since I do need it, I've been working on build_shlib and install_shlib distutils commands for building and installing shared libraries that extensions can link to. The commands are working on Linux, OS X 10.4, and (sort-of) on SGI Irix. I have a few questions: Why does UnixCCompiler._compile spawn self.compiler_so? This seems to be wrong (even for regular extensions). It works satisfactorily when the C compiler can also handle C++ code, but fails when the C and C++ compilers are separate, as when using the SGI MIPSpro compiler. Shouldn't _compile use CCompiler.detect_language and spawn compiler_cxx if necessary? (I think this is a problem not just with build_shlib, but with build_ext and build_clib as well.) py2app is the only other package I know that adds commands to distutils. It modifies the Distribution class like this: class Distribution(distutils.dist.Distribution): def __init__(self, attrs): etc... distutils.dist.Distribution.__init__(self, attrs) distutils.core.Distribution = Distribution The problem with this is that any other package (such as mine) that also modifies Distribution will either overwrite or be overwritten by the py2app modifications. Shouldn't Distribution be modified like this: oldDist = distutils.dist.Distribution class Distribution(oldDist): def __init__(self, attrs): etc... oldDist.__init__(self, attrs) distutils.dist.Distribution = Distribution Then if two different packages both modify the same class, the __init__ calls are chained together. What's the best way of testing my code on a variety of platforms? I only have easy access to a few different architectures and operating systems. There are spots in the code where I've pretty much just had to guess at what to do, based on other parts of distutils. Would someone who's more familiar with distutils than I am like to take a look at what I've done and give me some feedback? I can either e-mail the files or put them on a web site. They're not really ready for release yet, I don't think. Thanks, Steve -- -- EMail: stephen.langer@nist.gov Phone: (301) 975-5423 -- -- WWW: http://math.nist.gov/mcsd/Staff/SLanger/ Fax: (301) 990-4127 -- -- Mail: NIST; 100 Bureau Drive -- Stop 8910; Gaithersburg, Md 20899-8910 -- -- "I don't think this will work. That's why it's science." -- -- Naomi Langer, 17 Feb 2003 --
On Oct 3, 2005, at 10:56 AM, Stephen Langer wrote:
Is it possible to create libBase portably with distutils? It's possible to do it on Linux by subclassing build_ext.build_ext and explicitly using self.compiler.compile() and self.compiler.link_shared_lib() to build the shared library before calling build_ext.build_ext.build_extensions(). But the same thing on Mac OS X only creates libBase.so, whereas I need it to create libBase.dylib.
I don't know if this is possible, although I'd guess it is not.
That's too bad. Is there a reason for it? I'd volunteer to work on modifying distutils so that it can build a .dylib, but I am not an expert on library formats. I don't really know the difference between a .so and a .dylib, except that one of them works and the other doesn't. Can someone point me in the direction of a good reference on the topic?
The most portable way is to not build a libBase at all. Build a Python extension that has all the functionality from libBase available in it as a data structure with function pointers, and get a reference to that from everything that depends on libBase functionality. Take a look at what Numeric's C API does, for example. -bob
On Oct 3, 2005, at 3:15 PM, Bob Ippolito wrote:
On Oct 3, 2005, at 10:56 AM, Stephen Langer wrote:
Is it possible to create libBase portably with distutils? It's possible to do it on Linux by subclassing build_ext.build_ext and explicitly using self.compiler.compile() and self.compiler.link_shared_lib() to build the shared library before calling build_ext.build_ext.build_extensions(). But the same thing on Mac OS X only creates libBase.so, whereas I need it to create libBase.dylib.
I don't know if this is possible, although I'd guess it is not.
That's too bad. Is there a reason for it? I'd volunteer to work on modifying distutils so that it can build a .dylib, but I am not an expert on library formats. I don't really know the difference between a .so and a .dylib, except that one of them works and the other doesn't. Can someone point me in the direction of a good reference on the topic?
The most portable way is to not build a libBase at all. Build a Python extension that has all the functionality from libBase available in it as a data structure with function pointers, and get a reference to that from everything that depends on libBase functionality. Take a look at what Numeric's C API does, for example.
Will such a scheme allow me to define a C++ base class in one module, and have other modules define derived classes? I want users to be able to define derived classes without having to recompile the whole program. I also want to be able to choose which parts of the program are loaded, so the base classes and some of the derived classes will be in one library, but other derived classes will be separate. Thanks. -- Steve -- -- EMail: stephen.langer@nist.gov Phone: (301) 975-5423 -- -- WWW: http://math.nist.gov/mcsd/Staff/SLanger/ Fax: (301) 990-4127 -- -- Mail: NIST; 100 Bureau Drive -- Stop 8910; Gaithersburg, Md 20899-8910 -- -- "I don't think this will work. That's why it's science." -- -- Naomi Langer, 17 Feb 2003 --
On Oct 4, 2005, at 10:46 AM, Stephen Langer wrote:
On Oct 3, 2005, at 3:15 PM, Bob Ippolito wrote:
On Oct 3, 2005, at 10:56 AM, Stephen Langer wrote:
Is it possible to create libBase portably with distutils? It's possible to do it on Linux by subclassing build_ext.build_ext and explicitly using self.compiler.compile() and self.compiler.link_shared_lib() to build the shared library before calling build_ext.build_ext.build_extensions(). But the same thing on Mac OS X only creates libBase.so, whereas I need it to create libBase.dylib.
I don't know if this is possible, although I'd guess it is not.
That's too bad. Is there a reason for it? I'd volunteer to work on modifying distutils so that it can build a .dylib, but I am not an expert on library formats. I don't really know the difference between a .so and a .dylib, except that one of them works and the other doesn't. Can someone point me in the direction of a good reference on the topic?
The most portable way is to not build a libBase at all. Build a Python extension that has all the functionality from libBase available in it as a data structure with function pointers, and get a reference to that from everything that depends on libBase functionality. Take a look at what Numeric's C API does, for example.
Will such a scheme allow me to define a C++ base class in one module, and have other modules define derived classes? I want users to be able to define derived classes without having to recompile the whole program. I also want to be able to choose which parts of the program are loaded, so the base classes and some of the derived classes will be in one library, but other derived classes will be separate.
I don't know enough about C++ implementation details to answer that.. try it and see if it works. It would definitely work for anything C or Objective-C based, but C++ is crazy so I don't know. -bob
participants (3)
-
Bob Ippolito -
Ronald Oussoren -
Stephen Langer