Beginer questions
Hi All, I am a beginner in Python and kind of confused on the subject of combining C++ and Python. I understand that there are 2 ways of interaction. We in our project are only interesting in the case that you call C++ classes and functions from Python, We would like to use C++ classes that are already written in C++ from Python script. We successfully built a shared library with C api, import it and call the function from Python, but when it comes to calling C++, we are completely in the dark and would appreciate any help in that area. Once we understand the subject we would be glad to help other to come up to speed as well. Here is our questions: 1. At this point we are trying to wrap the C++ class as C, but our problem is, how do we pass a pointer from Python to C++ so we can keep the pointer to the class. 2. Is there any easer way to call the functions of C++ classes from python instead of wrapping them with C functions. 3. We recently came across CObject Python object. is it the solution to pass pointer to C function from Python. Is there any example available on this? 4. About Swig. Is swig the best solution for calling C++ from Python. Thanks a lot in advanced for any information which can shade some light on this subject. Ephi.
I just spent the last few days developing a C++ extension to Python. While I am by no means an expert, I do have some relevant information floating about in my short term memory. The extension we developed was much like what you describe. A rather complex C++ class (set of classes) is already in the form of a library, and we want to be able to instantiate these object(s) from Python, call their methods, and get at their data, etc... One of the programmers in my group first attempted this using the LLNL CXX package to develop the extension. This resulted in lots of trouble for us, which has yet to be resolved. Not because the CXX package isn't good. The problem may be that it is too good, or more precisely, uses C++ constructs that are too much for g++ (GCC-2.95.2). In this particular instance, g++ (Solaris) has a nasty bug that does not allow code built with it to catch exceptions. Instead, the exceptions get thrown all the way to the default exception handler which causes a core dump. We have absolutely no problems when we build the extension on Windows with VC++. Solaris, with g++ is another story altogether though. So, as a means to explore alternatives, I wrote the C++ extension by hand. It turned out to fairly simple, and efficient (at least so far, I am only part way through my unit tests). To do this, I just developed a type extension for Python, almost exactly as described in the book: "Internet Programming with Python" (see the bstreammodule example). Adding a new type to Python gives you the opportunity to stash a pointer to your C++ object in the object structure. Important things to note: 1) the init<module> function must be declared extern "C" so that it can be runtime linked with Python. The rest of the functions don't matter, as they are declared static, and they show up in the method tables anyways. 2) No global objects as we are linking against a C compiled Python. The C++ library the we wrote an extension for makes _heavy_ use of C++ STL containers. Getting select data out of the containers and into Python tuples, lists, and dictionaries is fairly simple. The hard part (assuming you are familiar with your own library) is learning the expectations of Python extensions. I found the book, and the standard Python documentation to be just fine. When we used the LLNL CXX extension, our contribution was about 500 lines of code. The "plain" extension is also about 500 lines of code. The main difference is in writing the plain extension I avoided use of C++ exceptions, and other C++ features that I know are not very portable (like new iostreams, but don't get me started...) I hope this helps. Cheers, -Ian Searle Ephi Dror wrote:
Hi All, I am a beginner in Python and kind of confused on the subject of combining C++ and Python. I understand that there are 2 ways of interaction. We in our project are only interesting in the case that you call C++ classes and functions from Python, We would like to use C++ classes that are already written in C++ from Python script. We successfully built a shared library with C api, import it and call the function from Python, but when it comes to calling C++, we are completely in the dark and would appreciate any help in that area. Once we understand the subject we would be glad to help other to come up to speed as well. Here is our questions:1. At this point we are trying to wrap the C++ class as C, but our problem is, how do we pass a pointer from Python to C++ so we can keep the pointer to the class.2. Is there any easer way to call the functions of C++ classes from python instead of wrapping them with C functions.3. We recently came across CObject Python object. is it the solution to pass pointer to C function from Python. Is there any example available on this?4. About Swig. Is swig the best solution for calling C++ from Python. Thanks a lot in advanced for any information which can shade some light on this subject. Ephi.
This reminds me to call attention of the C++ SIG to the article in the July/Aug 1999 C++ report that explains that pointers to C functions and pointers to C++ functions are no longer type compatible. This has serious implications for extending Python with C++, in that Python's tables expect a C function. Up to now it has been possible to just cast whatever to PyCFunction and move on; evidently some compilers will now start to object to that if the function in question is a C++ one. I haven't had time to look at this to see if I have a decent work-around.
on 11/14/99 20:54, Paul F. Dubois at dubois1@llnl.gov wrote:
This reminds me to call attention of the C++ SIG to the article in the July/Aug 1999 C++ report that explains that pointers to C functions and pointers to C++ functions are no longer type compatible. This has serious implications for extending Python with C++, in that Python's tables expect a C function. Up to now it has been possible to just cast whatever to PyCFunction and move on; evidently some compilers will now start to object to that if the function in question is a C++ one. I haven't had time to look at this to see if I have a decent work-around.
If it is a static function couldn't you just declare it as " extern "C" "??? I know that works fine for various callbacks in the OS. Jon.
Jonathan Wight wrote:
on 11/14/99 20:54, Paul F. Dubois at dubois1@llnl.gov wrote:
This reminds me to call attention of the C++ SIG to the article in the July/Aug 1999 C++ report that explains that pointers to C functions and pointers to C++ functions are no longer type compatible. This has serious implications for extending Python with C++, in that Python's tables expect a C function. Up to now it has been possible to just cast whatever to PyCFunction and move on; evidently some compilers will now start to object to that if the function in question is a C++ one. I haven't had time to look at this to see if I have a decent work-around.
If it is a static function couldn't you just declare it as " extern "C" "??? I know that works fine for various callbacks in the OS.
Jon.
Sorry for my ignorance. Is the C++ report something put out by the standard committee ? Is this something new to the standard, or has this "feature" been there for a while ? Thanks, -Ian
Ian R. Searle writes:
Sorry for my ignorance. Is the C++ report something put out by the standard committee ?
The C++ Report is a C++ trade rag that has been in circulation for about a decade, give or take. It is not affiliated with the committee, but it has over the yeards, reported on the actions of the committee. An excellent source for C++ trade lore.
Is this something new to the standard, or has this "feature" been there for a while ?
As far as I know, global C++ functions (non member functions) that are not explicitly declared to be extern "C", have never been guaranteed to be compatible with C functions. The impact of this on the Python/C++ integration work, imo, is that we must be able to build extension object method tables without casting to PyCFunction. This probably means implicitly-instantiable per-class templated trampoline machinery. Note, I had decided to not do the "cast to PyCFunction" thing on my first reading of the bstreammodule code. The CPython implementation uses PyCFunctions for two different function signatures--those with keyword args and those without. Clearly you can only get away with shenanigans like that in the unsafe world of C. Any C++ binding will have to regard such function signatures as being of different function pointer types, and strictly conforming C++ compilers will not let you blithely cast between them. -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski
Jonathan Wight writes:
on 11/14/99 20:54, Paul F. Dubois at dubois1@llnl.gov wrote:
This reminds me to call attention of the C++ SIG to the article in the July/Aug 1999 C++ report that explains that pointers to C functions and pointers to C++ functions are no longer type compatible. This has serious implications for extending Python with C++, in that Python's tables expect a C function. Up to now it has been possible to just cast whatever to PyCFunction and move on; evidently some compilers will now start to object to that if the function in question is a C++ one. I haven't had time to look at this to see if I have a decent work-around.
If it is a static function couldn't you just declare it as " extern "C" "??? I know that works fine for various callbacks in the OS.
You can declare a non-member function in C++ to have extern "C" linkage, whether it is static or not. Of course you lose linkage safety if you do this, but you can do it. -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski
Paul F. Dubois writes:
This reminds me to call attention of the C++ SIG to the article in the = July/Aug 1999 C++ report that explains that pointers to C functions and = pointers to C++ functions are no longer type compatible. This has = serious implications for extending Python with C++, in that Python's = tables expect a C function. Up to now it has been possible to just cast = whatever to PyCFunction and move on; evidently some compilers will now = start to object to that if the function in question is a C++ one. I = haven't had time to look at this to see if I have a decent work-around.
I ran into this problem with SWIG when I upgraded to a new C++ compiler. To get around it, I just placed all of the wrapper code inside an extern "C" block. I'm not sure if this is an ideal solution and I have no idea what it means for CXX. It seemed to work for SWIG however. Cheers, Dave
You can place all of the C++ function declarations, the ones that appear in the method tables in an extern "C" wrapper too. But, that has the very ugly side effect of exposing your entire interface to the world. -Ian David Beazley wrote:
Paul F. Dubois writes:
This reminds me to call attention of the C++ SIG to the article in the = July/Aug 1999 C++ report that explains that pointers to C functions and = pointers to C++ functions are no longer type compatible. This has = serious implications for extending Python with C++, in that Python's = tables expect a C function. Up to now it has been possible to just cast = whatever to PyCFunction and move on; evidently some compilers will now = start to object to that if the function in question is a C++ one. I = haven't had time to look at this to see if I have a decent work-around.
I ran into this problem with SWIG when I upgraded to a new C++ compiler. To get around it, I just placed all of the wrapper code inside an extern "C" block. I'm not sure if this is an ideal solution and I have no idea what it means for CXX. It seemed to work for SWIG however.
Cheers,
Dave
Ian Searle writes:
One of the programmers in my group first attempted this using the LLNL CXX package to develop the extension. This resulted in lots of trouble for us, which has yet to be resolved. Not because the CXX package isn't good. The problem may be that it is too good, or more precisely, uses C++ constructs that are too much for g++ (GCC-2.95.2). In this particular instance, g++ (Solaris) has a nasty bug that does not allow code built with it to catch exceptions. Instead, the exceptions get thrown all the way to the default exception handler which causes a core dump. We have absolutely no problems when we build the extension on Windows with VC++. Solaris, with g++ is another story altogether though.
I have also been unable to use GCC-2.95 for my other code projects, due to other apparent code gen bugs and miscelaneous other problems. GCC 2.95 is a landmark event--a reason for great excitement--but it isn't prime-time yet, I'm afraid.
So, as a means to explore alternatives, I wrote the C++ extension by hand. It turned out to fairly simple, and efficient (at least so far, I am only part way through my unit tests). To do this, I just developed a type extension for Python, almost exactly as described in the book: "Internet Programming with Python" (see the bstreammodule example). Adding a new type to Python gives you the opportunity to stash a pointer to your C++ object in the object structure. Important things to note:
Right, this is a very feasible approach. The goal of CXX is to make this as easy as declaring a derived object. But one can certainly do it oneself.
2) No global objects as we are linking against a C compiled Python.
There is a patch posted on the C++ SIG website, which modifies the configure script for Python to allow you to get your main linked with C++, which ensoures that global ctors get run correctly. If you use that, you don't need rule 2) above.
The C++ library the we wrote an extension for makes _heavy_ use of C++ STL containers.
Note that this is in potential contradiction with your rule 2) above. I'm sure you know that iostreams depends on global ctors, so your rule 2) means that cout is out. But also, some STL implementations make use of static global objects, and I don't see anything to imply that such an implementation is specifically nonconformant. So I would be worried that your current prescription is working due to fortuitous happenstance, depending on unspecified behavior of your current tool chain(s).
Getting select data out of the containers and into Python tuples, lists, and dictionaries is fairly simple. The hard part (assuming you are familiar with your own library) is learning the expectations of Python extensions. I found the book, and the standard Python documentation to be just fine.
CXX is also supposed to make this issue of ownership management trivial.
When we used the LLNL CXX extension, our contribution was about 500 lines of code. The "plain" extension is also about 500 lines of code. The main difference is in writing the plain extension I avoided use of C++ exceptions, and other C++ features that I know are not very portable (like new iostreams, but don't get me started...)
Frustrating, isn't it? I would add though, that its been more than 3 years since I have personally experienced a vendor C++ compiler whose exception mechanism didn't work. Outside of GCC, the rough points in C++ these days seem to lie in the areas of partial specialization, namespace resolution lookup rules, and associated arcana. GCC is generally conformant in almost all areas, but just has some issues with uneven implementation quality. Best regards, -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski
On Monday 15 November, Geoffrey Furnish wrote: [...]
2) No global objects as we are linking against a C compiled Python.
There is a patch posted on the C++ SIG website, which modifies the configure script for Python to allow you to get your main linked with C++, which ensoures that global ctors get run correctly. If you use that, you don't need rule 2) above.
I'm intrigued by the assertion that you can't use global objects when the Python executable is compiled with C. For which platforms is it true? I've successfully used global C++ objects with constructors on Linux, Solaris and Windows NT, without compiling Python itself with C++. I've done this while providing Python bindings for omniORB. My Python extension module does not have any global objects itself, but it is (dynamically) linked with the main omniORB library which does. Maybe that's why it works? Cheers, Duncan. -- -- Duncan Grisby \ Research Engineer -- -- AT&T Laboratories Cambridge -- -- http://www.uk.research.att.com/~dpg1 --
Duncan Grisby writes:
On Monday 15 November, Geoffrey Furnish wrote:
[...]
2) No global objects as we are linking against a C compiled Python.
There is a patch posted on the C++ SIG website, which modifies the configure script for Python to allow you to get your main linked with C++, which ensoures that global ctors get run correctly. If you use that, you don't need rule 2) above.
I'm intrigued by the assertion that you can't use global objects when the Python executable is compiled with C. For which platforms is it true? I've successfully used global C++ objects with constructors on Linux, Solaris and Windows NT, without compiling Python itself with C++.
1. ((off topic) If you eat lunch with anyone who's responsible for http://www.uk.research.att.com/vnc, pass on my thanks for a great piece of software) 2. I've been assuming that all implementations that use ELF-based shared libraries initialize global objects correctly. This was mentioned in an article in Computing in Science and Engineering: @Article{Gray99Shadow, author = {M. C. Gray and R. M. Roberts and T. M. Evans}, title = {Shadow-object interface between Fortran95 and C++}, journal = {Comp. in Sci. and Engineer.}, year = 1999, volume = 1, number = 2, pages = {63-70}, annote = {http://computer.org} } They support their statment with a footnote to the Linker and Libraries Guide, Solaris 2.6 Software Developer Collection vol 1., http:://docs.sun.com. Phil
Duncan Grisby writes:
2) No global objects as we are linking against a C compiled Python.
There is a patch posted on the C++ SIG website, which modifies the configure script for Python to allow you to get your main linked with C++, which ensoures that global ctors get run correctly. If you use that, you don't need rule 2) above.
I'm intrigued by the assertion that you can't use global objects when the Python executable is compiled with C. For which platforms is it true?
C does not have global ctors, so if you build a C program, you can't expect C++ global ctors to get run, unless you have detailed, platform-specific knowledge to the contrary. Exactly how global ctors get run in C++ is platform specific. The standard just mandates that global ctors get run before main() is called. Exactly how it happens depends on the compiler. I think Myers explains this in his first book at some length.
I've successfully used global C++ objects with constructors on Linux, Solaris and Windows NT, without compiling Python itself with C++.
You were lucky. That's the linker being nice to you.
I've done this while providing Python bindings for omniORB. My Python extension module does not have any global objects itself, but it is (dynamically) linked with the main omniORB library which does. Maybe that's why it works?
Many dynamic linkers do the global ctor thing correctly. BTW, this is not just ELF as was mentioned in a sperate post. In general, a good dynamic linker ought to understand the notion that a dynloaded library might need initialization or finalization code to be run. But that's part of the ABI for a computing platform, not the C++ language definition. On the other hand, if you build a Python with a C++ extension module statically linked in, it will quite likely fail, unless the C++ compiler compiles main(), and drives the link. In this case, the issue is not just getting the special initialization and finalization code to be draped around main(), but also because of the need to perform implicit instantiations. Again, one can luck out. Some systems have linkers that "do it right" irrespective of what the compilers did. And some compilers use gross bloating template instantiation techniques that ensure simplistic linkage models will "work", but do so at the expense of bloating executable size, and consequent reductions in I-cache locality. But if you want it to work because you did it right, and not merely because you were lucky, then you need to compile main() with C++, and let C++ run the link. -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski
Geoffrey Furnish writes:
But if you want it to work because you did it right, and not merely because you were lucky, then you need to compile main() with C++, and let C++ run the link.
--
Further to that goal, patch.geoff is failing for me on Makefile.pre.in and configure.in. To get it to work I had to cp Modules/Makefile.pre.in to Python-1.5.2 and replace the line AC_REVISION($Revision: 1.103 $) in configure.in with AC_REVISION($Revision: 1.1 $) Regards, Phil
Ian already mentioned the possibility of writing your own type extension. Building on that notion... Ephi Dror writes:
1. At this point we are trying to wrap the C++ class as C, but our problem is, how do we pass a pointer from Python to C++ so we can keep the pointer to the class.
You can arrange that the methods of your type extension downcast the "self" pointer passed to them, to make it a pointer to your object, and then invoke member functions on it. Something like this: static PyObject * myext_meth1( PyObject *self, PyObject *args ) { MyObject *myo = static_cast<MyObject *>(self); return myo->meth1( args ); }
2. Is there any easer way to call the functions of C++ classes from python instead of wrapping them with C functions.
You can write scripts to automate the production of wrappers like that shown above. Or you can use CXX, as Ian mentioned. Some formalization of the techniques exhibited in CXX is what several of us regard as the end goal deliverable for this SIG. But it is proving challenging to do this, given the constraints of personal time and effort, as well as standards conformance. -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski
participants (9)
-
David Beazley -
Duncan Grisby -
Ephi Dror -
Geoffrey Furnish -
Ian R. Searle -
Ian Searle
-
Jonathan Wight -
Paul F. Dubois -
Phil Austin