How can I have a global C/C++ variable that is shared between Python C/C++ extensions?
Dear list, Apologies if this question is slightly off-topic, but I thought I might get answers here rather than the other Python mailing lists. I have a rather specific issue with creating a Python C++ extension that I'm really stuck on it. These are the details... -- I have written a Python package almost completely in C++. The reason for doing so is because I want to manually wrap an existing C++ library, but that is irrelevant here. This Python package consists of a number of different extension modules, all of which I compile with distutils in a 'setup.py' script. These extension modules can be interrelated, in which case I link them by passing the shared library to the Extension constructor. To be clear, suppose I have two Python C++ modules, A and B, where B uses functions defined in A. These normally compile into A.so and B.so. Since B uses functions defined in A, I compile the A module as usual then I pass ':A.so' as a library to the libraries keyword in the Extension constructor for the B module. (The ':' lets g++ deal with the fact that the library does not start with the usual 'lib' prefix.) This works fine for linking functions and classes. My problem is as follows: I have defined some extern global C++ variables in A. While doing what I have described allows B to access functions in A, it actually seems to create a COPY of any global data defined in A. This is a real problem for me. It seems to me that the issue is essentially similar to having global variables across shared libraries. The solutions to this that I have found online do not seem to solve the problem. Any help would be so very appreciated!
I had a similar problem when testing my extensions on OSX. I guess you're not on OSX, but if you test out on it, I'm sure you'd find even more linking problems.. Problem is that the bundle format - the default Python extension format, on OSX - doesn't allow compile-time linking, only run-time dynamic loading. Python's distutils is quite tricky when it comes to making shared libraries on OSX, or dylibs, but I found them necessary, when sharing symbols between extensions. YMMV, but I what I did was to create an extension (a file with only `BOOST_PYTHON_MODULE(foo)`, to be dlopen'd by Python's import machinery) and separate, shared libraries, to be linked in at compile time. The folder structure I end up with, looks a little like this:- pkg/__init__.py pkg/A.py pkg/B.py pkg/_A.so pkg/_B.so pkg/lib/libA.dylib pkg/lib/libB.dylib Then, you don't to link across extensions, only to libraries, which should work as normal. I did have to add OS-specific runtime_library_flags, which distutils was having problems generating itself. I haven't tested on Windows, but, these are some of the exact options I use, when making distutils Extension instances. For an Extension (e.g. _A.so): # OS-dependent flags for dynamic symbol resolution: libraries = ["A"] library_dirs = [ <path to build directory>, ... ] if sys.platform.startswith('linux'): runtime_library_dirs += ["${ORIGIN}/lib"] elif sys.platform == 'darwin': extra_link_args = ['-Wl,-rpath,@loader_path/', '-install_name', '_A.so', '-compatibility_version', __version__, '-current_version', __fullversion__ ] elif sys.platform == 'win32': runtime_library_dirs += ['lib-dynload'] And then for a library (e.g. libA.dylib):- # OS-dependent flags for dynamic symbol resolution: libraries = ["B"] library_dirs = [ <path to build directory> , ...] if platform == "linux2": extra_link_args = ["-Wl,-soname,libA.so"] elif sys.platform == 'darwin': extra_link_args = [ '-install_name', '@rpath/lib/libA.dylib', '-Wl,-rpath,@loader_path/..', '-compatibility_version', __version__, '-current_version', __fullversion__] HTH, Alex On Wed, 12 Jun 2013 23:23:53 +0100, octalber fest <octalberfest@gmail.com> wrote:
Dear list,
Apologies if this question is slightly off-topic, but I thought I might get answers here rather than the other >Python mailing lists. I have a rather specific issue with creating a Python C++ extension that I'm really stuck on >it. These are the details...
--
I have written a Python package almost completely in C++. The reason for doing so is because I want to manually >wrap an existing C++ library, but that is irrelevant here.
This Python package consists of a number of different extension modules, all of which I compile with distutils in a >'setup.py' script. These extension modules can be interrelated, in which case I link them by passing the shared >library to the Extension constructor. To be clear, suppose I have two Python C++ modules, A and B, where B uses >functions defined in A. These normally compile into A.so and B.so. Since B uses functions defined in A, I compile >the A module as usual then I pass ':A.so' as a library to the libraries keyword in the Extension constructor for >the B module. (The ':' lets g++ deal with the fact that the library does not start with the usual 'lib' prefix.) >This works fine for linking functions and classes.
My problem is as follows: I have defined some extern global C++ variables in A. While doing what I have described >allows B to access functions in A, it actually seems to create a COPY of any global data defined in A. This is a >real problem for me.
It seems to me that the issue is essentially similar to having global variables across shared libraries. The >solutions to this that I have found online do not seem to solve the problem.
Any help would be so very appreciated!
-- Using Opera's mail client: http://www.opera.com/mail/
participants (2)
-
Alex Leach -
octalber fest