Hi Mark, Thanks a lot again. We tried it but unfortunately it did not help in our case. Does anyone know where can we download C++ based debug version of Python 1.52 for Linux? Our next step is to use Python 1.52 which is compiled with C++ and possibly a debug version and then try our little sample. Thanks a lot. Ephi. -----Original Message----- From: Mark Hammond [mailto:mhammond@skippinet.com.au] Sent: Wednesday, November 24, 1999 12:53 AM To: 'Ephi Dror'; c++-sig@python.org Subject: RE: [C++-SIG] Python calling C++ issues
Unfortunately, we don't seems to understand what do you mean by "Object". Is it something we need to define or is it already defined in a h file that we forgot to include. We use Linux/gcc environment.
Sorry - I meant "PyObject"
Also, in our sample, we did not use virtual function
Hmmm - OK - my advice could well be wrong. Unless your C compiler always adds a vfptr. I have seen this problem (when using virtuals) before and the symptoms sound almost identical. Try the code snippet I posted, and see if the debugger does show the "PyObject *" and the "Spam *" as different values, even though they are pointing to the same object. Otherwise Im afraid I have no idea... Mark. _______________________________________________ C++-SIG maillist - C++-SIG@python.org http://www.python.org/mailman/listinfo/c++-sig
Hi Mark,
Thanks a lot again. We tried it but unfortunately it did not help in our case.
Does anyone know where can we download C++ based debug version of Python 1.52 for Linux?
Our next step is to use Python 1.52 which is compiled with C++ and
Hi All, Thanks a lot for all the support we got so far. Unfortunately, as of this moment, we have not yet successfully run our simple spam program. It is partially worked but then we got the two unexplained strange phenomena that as of now, nobody seems to understand why. Also, even if we got those problems solved using compiler xyz version 123 who knows what other problems are waiting for us in the next corner. It may work perfect on NT, Win98 , SUN, etc platforms but we need it to work on RedHat 6.1 using the distributed Python 1.5.2 and GNU as it come out of the box when you install Linux 6.1. For us, it just does not work. I am personally very disappointed from all of this. I really truly wanted to give python a chance. I got the impression by reading one of the Python books back cover that what we want to do is common practice and should be a plug and play . (http://www.amazon.com/exec/obidos/ts/book-reviews/1565921976/103-5409830-17 11020) Reading the books give the impression that Python and C++ were meant to live in peace and be haply working with each other. Unfortunately, my personal experience realty proved the opposite. Python may work nicely with C code, but who needs that 30 days before the year 2000 !!, Programmers today need at least integration with existing C++ code and very soon they need integration with Java code. Our example only had the simplest C++ class in the word, and Python could not even use it correctly. Is just un believable to me. It is possible that we made stupid mistakes in the code, it is also possible that we are bad programmers. Who knows?. We did not hide our code. In fact, I attached it to my original email. Does anyone know how much Python is really used on Redhat Linux with the conjunction with C++, especially the direction of Python calling C++ functions? We have not given up yet but we are close. I would love to hear from the group that I am wrong. That I was just blind and simply did not get it. That calling C++ class from Python is really a piece of cake after all and especially on RedHat 6.1 Linux with standard installation components. Thank you all for helping us. We hope to stay in the group and contribute in the future, but it all depend on what next for us in terms of using Python/C++. Ephi. ----- Original Message ----- From: Dror, Ephi <ephi.dror@cis.canon.com> To: <mhammond@skippinet.com.au>; <c++-sig@python.org> Sent: Wednesday, November 24, 1999 10:46 AM Subject: RE: [C++-SIG] Python calling C++ issues possibly
a debug version and then try our little sample.
Thanks a lot.
Ephi.
-----Original Message----- From: Mark Hammond [mailto:mhammond@skippinet.com.au] Sent: Wednesday, November 24, 1999 12:53 AM To: 'Ephi Dror'; c++-sig@python.org Subject: RE: [C++-SIG] Python calling C++ issues
Unfortunately, we don't seems to understand what do you mean by "Object". Is it something we need to define or is it already defined in a h file that we forgot to include. We use Linux/gcc environment.
Sorry - I meant "PyObject"
Also, in our sample, we did not use virtual function
Hmmm - OK - my advice could well be wrong. Unless your C compiler always adds a vfptr.
I have seen this problem (when using virtuals) before and the symptoms sound almost identical. Try the code snippet I posted, and see if the debugger does show the "PyObject *" and the "Spam *" as different values, even though they are pointing to the same object.
Otherwise Im afraid I have no idea...
Mark.
_______________________________________________ C++-SIG maillist - C++-SIG@python.org http://www.python.org/mailman/listinfo/c++-sig
_______________________________________________ C++-SIG maillist - C++-SIG@python.org http://www.python.org/mailman/listinfo/c++-sig
Ephi, I'd suggest you take a look at SWIG - the simplified wrapper interface generator. I've had great success in using it to wrap the C++ cross-platform GUI FLTK. If you need static constructors to run, I think you can use the %init statment somehow, even with a C-based Python interpreter. Python is all people say it is. The only thing I don't like about it compared to straight C++ is that it is not easy to provide users with a single run-time executable - but then they should have Python installed anyway! :)
Reading the books give the impression that Python and C++ were meant to live in peace and be haply working with each other.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Sr Software Engineer, Morse Controls http://www.geocities.com/SiliconValley/Peaks/4673 "Do or do not. There is no 'Try.' " -Yoda
Unfortunately, as of this moment, we have not yet successfully run our simple spam program.
You give up too easily... :) Assuming you haven't changed the source from the original post: The problem lies in the differences between Python extension modules and Python extension types. You have written an extension module, but are returning the values as if it were a type. Here's how to fix it: 1. The file c_spam.cpp is pointless, so get rid of it. 2. In ext_spam.cxx: static PyObject* ext_new_Spam(PyObject* self, PyObject* args) { int a, b; if (!PyArg_ParseTuple(args, "ii", &a, &b)) return NULL; Spam* spam = new_Spam(a, b); return Py_BuildValue("O", spam); } 'Spam' is a C++ object, not a python object, and you can't just cast it to a PyObject* and expect it to work. One way to do this is to pass back a pointer to the object as a long: static PyObject* ext_new_Spam(PyObject* self, PyObject* args) { int a, b; if (!PyArg_ParseTuple(args, "ii", &a, &b)) return NULL; Spam* spam = new Spam(a, b); return Py_BuildValue("l", (long) spam); // Or return PyInt_FromLong((long) spam); } The Python script now stores this as a Python integer object. 3. In all other functions in ext_spam.cxx: e.g. static PyObject* ext_Spam_Display(PyObject* self, PyObject* args) { Spam* spam; if (!PyArg_ParseTuple(args, "O", &spam)) return NULL; Spam_Display(spam); return Py_BuildValue(""); } Use: static PyObject* ext_Spam_Display(PyObject* self, PyObject* args) { Spam* spam; if (!PyArg_ParseTuple(args, "l", &spam)) return NULL; Spam_Display(spam); Py_INCREF(Py_None); // Use these lines when you don't return Py_None; // want to return a value } That should just about fix it. A couple of other things: o If you don't already have Mark Lutz's book "Programming Python", get yourself a copy, and have a look through the chapters on embedding and extending Python. o If you want a better response, try posting to the main Python mailing list, not the C++ binding one. This list is more to do with the procedures for automating the process of creating Python objects which mirror C++ objects. o There are ways to do this sort of thing in a much more groovy way. Judging by the test script: #!/usr/bin/python import c_spam myspam=c_spam.new_Spam(8, 5) c_spam.Spam_Display(myspam) c_spam.delete_Spam(myspam) print "python end" ...you are using it like a module, not a type. There are ways to get Python to make your Spam object usable like this: import c_spam myspam = c_spam.Spam(8, 5) myspam.Display() del myspam I don't really want to give too much away, then you'll never learn how to do things yourself, but if you need any more assistance, don't hesistate to mail back. Good luck. -------------------------------------------- Adrian Eyre <mailto:a.eyre@optichrome.com> Optichrome Computer Solutions Ltd Maybury Road, Woking, Surrey, GU21 5HX, UK Tel: +44 1483 740 233 Fax: +44 1483 760 644 http://www.optichrome.com --------------------------------------------
On Wednesday 1 December, "Adrian Eyre" wrote: [...]
'Spam' is a C++ object, not a python object, and you can't just cast it to a PyObject* and expect it to work. One way to do this is to pass back a pointer to the object as a long:
static PyObject* ext_new_Spam(PyObject* self, PyObject* args) { int a, b;
if (!PyArg_ParseTuple(args, "ii", &a, &b)) return NULL;
Spam* spam = new Spam(a, b); return Py_BuildValue("l", (long) spam); // Or return PyInt_FromLong((long)
I haven't particularly followed the discussion, but I just want to point out that doing what you suggest is extremely dangerous. There is no guarantee that longs are the same size as pointers. There is also nothing to stop the Python code from modifying the integer, causing the C code to dereference arbitrary memory locations. It shouldn't be possible for Python code to cause core dumps. You should use PyCObjects instead. Cheers, Duncan. -- -- Duncan Grisby \ Research Engineer -- -- AT&T Laboratories Cambridge -- -- http://www.uk.research.att.com/~dpg1 --
I haven't particularly followed the discussion, but I just want to point out that doing what you suggest is extremely dangerous. There is no guarantee that longs are the same size as pointers.
There's no guarantee of most types in C++, only that a long is at least 1 byte. It's up to the programmer to make their C++ code platform independent by using #ifdefs and typedefs, but I thought that much was obvious. Besides which, the criteria IIRC was to run in on a PC on Linux, for which the stated code will work.
There is also nothing to stop the Python code from modifying the integer, causing the C code to dereference arbitrary memory locations.
Also true. However, if providing the Python programmer is aware of how the extension works, and then chooses to modify the pointer, or whatever, then they Deserve To Lose (tm).
It shouldn't be possible for Python code to cause core dumps. You should use PyCObjects instead.
It shouldn't, no, but I present you with the following code: #!/usr/bin/env python l = [] m = [] l.append(m) m.append(l) print l == m It was just a reponse to get the code to work. I mentioned that there were better ways of doing it. Although interestingly enough, code almost identical to what I posted appears in Mark Lutz's Programming Python book, as a valid way of achieve an extension type/class. -------------------------------------------- Adrian Eyre <mailto:a.eyre@optichrome.com> Optichrome Computer Solutions Ltd Maybury Road, Woking, Surrey, GU21 5HX, UK Tel: +44 1483 740 233 Fax: +44 1483 760 644 http://www.optichrome.com --------------------------------------------
There's no guarantee of most types in C++, only that a long is at least 1 byte.
From K & R, 2nd ed., 2:2, verse 8, p. 36:
"Each compiler is free to choose appropriate sizes for its own hardware, subject only to the restriction that shorts and ints are at least 16 bits, longs are at least 32 bits, and short is no longer than int, which is no longer than long." Everyone may be seated now... Cheers, Dave
From K & R, 2nd ed., 2:2, verse 8, p. 36:
"Each compiler is free to choose appropriate sizes for its own hardware, subject only to the restriction that shorts and ints are at least 16 bits, longs are at least 32 bits, and short is no longer than int, which is no longer than long."
That's rather annoying. What if my processor only has 1-bit buses, and my RAM only stores 10 words. I could never have any integers. :( -------------------------------------------- Adrian Eyre <mailto:a.eyre@optichrome.com> Optichrome Computer Solutions Ltd Maybury Road, Woking, Surrey, GU21 5HX, UK Tel: +44 1483 740 233 Fax: +44 1483 760 644 http://www.optichrome.com --------------------------------------------
David Beazley wrote:
There's no guarantee of most types in C++, only that a long is at least 1 byte.
From K & R, 2nd ed., 2:2, verse 8, p. 36:
"Each compiler is free to choose appropriate sizes for its own hardware, subject only to the restriction that shorts and ints are at least 16 bits, longs are at least 32 bits, and short is no longer than int, which is no longer than long."
Those limits do NOT apply to C++: there is NO requirement on the number of bits, except that it be documented by the vendor. [The comparative length requirements apply] The conversion of pointers to ints is not supported in C++: more precisely, the cast MAY be implemented by the vendor, if so, the vendor must document it, and it MUST hold the whole pointer so that: (void*)(long)(void*) p == p For those using C Python sources with C++, there are two gotchas. 1) Enumerations are NOT integers in C++ [Luckily, Guido doesn't use them much] 2) The very core of Python contains a serious breach of C++ rules: it casts functions pointers, which is NOT allowed AT ALL. [I'm responsible for that rule!] Luckily, this actually works on most C++ compilers. To Guido's credit, it is the only breach of strict ISO C++ Standards compliance I'm aware of. [In fact, there is a way around it .. but Guido has chosen not to use that technique] If I can give a summary: the Python core language source code is in EXCELLENT shape as a C++ program. -- John Skaller, mailto:skaller@maxtal.com.au 10/1 Toxteth Rd Glebe NSW 2037 Australia homepage: http://www.maxtal.com.au/~skaller voice: 61-2-9660-0850
skaller writes:
David Beazley wrote: The conversion of pointers to ints is not supported in C++:
Right. Pointer values must be held in objects of pointer type. And anticipating the following, I'll go ahead and state that pointer values must be held in objects of pointer type which are of the same type as the pointer in question (or which are accessible through valid pointer conversions).
For those using C Python sources with C++, there are two gotchas.
1) Enumerations are NOT integers in C++ [Luckily, Guido doesn't use them much]
Just to amplify, you can always make an int from an enum if you need to. A common case where this issue comes up, is in situations like the following: enum { This, That }; vector<int>::iterator it = find( v.begin(), v.end(), That ); which will not work (will not compile), but which can be repaired by doing: vector<int>::iterator it = find( v.begin(), v.end(), int(That) );
2) The very core of Python contains a serious breach of C++ rules: it casts functions pointers, which is NOT allowed AT ALL.
[I'm responsible for that rule!]
Thank you!
Luckily, this actually works on most C++ compilers.
I disagree with this. It does not work on /any/ "C++" compilers. There are however, on the market, many compilers which support any of a family of as yet undocumented languages, which seem all to be similar to "C++" in many ways, but which evidently support various "extension" constructs that are illegal in "C++", such as casting pointers. The right thing to do, is to avoid efforts to cast function pointers. For the Python/C++ interface, this basically means you need a seperate extension type object for each C++ object you wish to register as a Python extension. Making this machinery work easily and maintainably seems to be challenging. I don't personally have much hope for doing it, except through heavy use of template mechanisms, which have been the source of much consternation on this sig. -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski
Geoffrey Furnish wrote:
The right thing to do, is to avoid efforts to cast function pointers.
For the Python/C++ interface, this basically means you need a seperate extension type object for each C++ object you wish to register as a Python extension.
I don't believe this is the case. See below.
Making this machinery work easily and maintainably seems to be challenging. I don't personally have much hope for doing it, except through heavy use of template mechanisms, which have been the source of much consternation on this sig.
I don't really understand this. It is very easy to do. In some kind of minor simplification of python, exactly one abstract base class is required. This class has a vtable (set of pure virtual functions) which match the Python vtable. That's it: all extension type derive from this abstract base, and they're all automatically valid python types. Now the gotcha: instead of a single linear vtable, python has several subtables. The presence or absence of the subtable pointer (e.g the pointer to the sequence subtable) has semantics. This complication may mean some tricky design is needed; perhaps a minor restriction of the extension type semantics (like: 'you can't be both a sequence and map'). I will give an example, by taking a tiny subset of the python virtual functions: the len function, for example: class cpp_py_object { virtual int len() const = 0; ... }; Now, by using THIS class as the sole base of all cpp extension types, you only need a single CPython type, with a pointer to the C++ object of type cpp_py_object. By FAR the most interesting aspect of this is that you can dynamically load shared libraries using dlopen (or windows equivalent), which have factory functions in them which create extension types. These DLL's don't [necessarily] need to know anything about the Python wrapper objects which wrap them. -- John Skaller, mailto:skaller@maxtal.com.au 10/1 Toxteth Rd Glebe NSW 2037 Australia homepage: http://www.maxtal.com.au/~skaller voice: 61-2-9660-0850
skaller writes:
Geoffrey Furnish wrote:
For the Python/C++ interface, this basically means you need a seperate extension type object for each C++ object you wish to register as a Python extension.
I don't believe this is the case. See below. [...] I will give an example, by taking a tiny subset of the python virtual functions: the len function, for example:
class cpp_py_object { virtual int len() const = 0; ... };
Now, by using THIS class as the sole base of all cpp extension types, you only need a single CPython type, with a pointer to the C++ object of type cpp_py_object.
Okay, great. So we can invoke the virtual function len() on an extension object of unknown type. Now what's the plan for invoking non virtual methods of C++ objects without knowing their type? -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski
Geoffrey Furnish wrote:
Okay, great. So we can invoke the virtual function len() on an extension object of unknown type.
yes.
Now what's the plan for invoking non virtual methods of C++ objects without knowing their type?
That is impossible. It is not necessary to be able to do this, since the Python parser cannot generate any code which calls it. Bytecode can only call 'virtual functions' on PyObjects. Perhaps, you mean, how do we do this: object.method(arguments) And the answer is: the same way python does. It is the only way. Python calls 'getattr' to fetch 'method' by name. The resulting function fetched has a fixed calling sequence. Otherwise the bytecode interpreter could never call it. It accepts a pair of arguments, a tuple and a dictionary. [Actually, there is a complication with an obsoleted flag, determining if **kwds is support -- but lets ignore that for clarity]. In other words, it is impossible to call ANYTHING from python OTHER than one of the virtual methods. And so that is all that needs to be provided. Now, this is not to say that C++ cannot call other C++ method directly. But you have to understand that a C++ method like void method (int x, float y) const cannot ever be called by Python. Here's the sequence of events: 1) Python calls the C 'getattr' method of the single, fixed C++ wrapper type: this is a C function. 2) The C function casts the object pointer to the abstract base type of the C++ heirarchy. 3) Then, it calls the C++ virtual getattr method, after suitably rearranging the arguments. 4) The C++ getattr method has to return something that can be converted into a function which Python can call; that is, it has to return a C function which takes a tuple and dictionary as an argument. It finds this function in a table. Because C is a bit brain dead, there is no way to create a closure here. [This is possible using Viper, since the implementation language, ocaml, supports them: generic extensions in Viper are much easier than in C or C++ [but binding existing C libraries is, conversely, harder] The C function takes the first element of the tuple argument, which is the object, and casts it to the CONCRETE type begin defined, and calls the corresponding C++ method, munging arguments as required. [Smly, the return value of the C++ method has to be converted to a PyObject for the C function to return] There are many ways to do this; using SWIG is one; but it is IRRELEVANT. You can write it by hand for example. The point is that only ONE abstract C++ class is required as a base, and only ONE Python type object is required to wrap it, and this WILL handle every single C++ class suitably derived from the abstract base. I hope this is clear: the lookup of specific non-virtual method on a class is independent of the _automatically_ polymorphic lookup of the various python virtual methods, including getattr and, as you mentioned, len. It makes sense, to have a specific derived class, in which the virtual getattr is overriden, and uses a particular data structure and algorithm to do attribute lookup; for example, for the kind of table SWIG or other automatic generator generates. If I were implementing a getattr method by hand, I'd be tempted to us an STL map, myself, since it is likely to MUCH faster than the standard linear search typically used implementing extension types. Anyhow, the point is that there is nothing new here, this is no different to C: finding attributes of objects, including methods, has nothing to do with the Python interface to the type object: it is universally achieved by a type specific getattr method. -- John Skaller, mailto:skaller@maxtal.com.au 10/1 Toxteth Rd Glebe NSW 2037 Australia homepage: http://www.maxtal.com.au/~skaller voice: 61-2-9660-0850
I for one am losing track of the argument here. Everything you say below about the mechanism that Python uses for finding and invoking functions of modules, is well known, and if you didn't read it anywhere first, can be directly observed from the Python C source. The part of your post which relates to a factual rendition of how things works in CPython today, is not in contention. Your claim that this can all be handled with a single base class is--until you demonstrate it by posting ocde which works as you posit--at best understood to be an hypothesis. Several of us on this list, including myself, Paul, Barry, and doubtless others, have worked out schemes largely similar to what you describe. What has not yet been done, is to field an implementation that both has the desired features, works with exactly one base class for all extensions, and is correct according to the C++ language standard. I have pursued a slightly different approach which is correct according to the language standard, but uses templates to fabricate on demand, much of the machinery which is neded to make all the funciton invocations. I am sorry that I have not yet been able to get this fully dusted off and whipped into presentable form. Still working on that. Meanwhile, Paul is pursing a system which does not have that characteristic, but instead works more like what you describe. But he is also not done, and still working out final details. I invite you to develop and post code which works as you claim can be done. We would all love to see a tangible incarnation of the design ideas you espouse. In other words, show us the beef. skaller writes:
Geoffrey Furnish wrote:
Okay, great. So we can invoke the virtual function len() on an extension object of unknown type.
yes.
Now what's the plan for invoking non virtual methods of C++ objects without knowing their type?
That is impossible. It is not necessary to be able to do this, since the Python parser cannot generate any code which calls it. Bytecode can only call 'virtual functions' on PyObjects.
Perhaps, you mean, how do we do this:
object.method(arguments)
And the answer is: the same way python does. It is the only way. Python calls 'getattr' to fetch 'method' by name. The resulting function fetched has a fixed calling sequence. Otherwise the bytecode interpreter could never call it. It accepts a pair of arguments, a tuple and a dictionary.
[Actually, there is a complication with an obsoleted flag, determining if **kwds is support -- but lets ignore that for clarity].
In other words, it is impossible to call ANYTHING from python OTHER than one of the virtual methods. And so that is all that needs to be provided.
Now, this is not to say that C++ cannot call other C++ method directly. But you have to understand that a C++ method like
void method (int x, float y) const
cannot ever be called by Python. Here's the sequence of events:
1) Python calls the C 'getattr' method of the single, fixed C++ wrapper type: this is a C function.
2) The C function casts the object pointer to the abstract base type of the C++ heirarchy.
3) Then, it calls the C++ virtual getattr method, after suitably rearranging the arguments.
4) The C++ getattr method has to return something that can be converted into a function which Python can call; that is, it has to return a C function which takes a tuple and dictionary as an argument.
It finds this function in a table. Because C is a bit brain dead, there is no way to create a closure here. [This is possible using Viper, since the implementation language, ocaml, supports them: generic extensions in Viper are much easier than in C or C++ [but binding existing C libraries is, conversely, harder]
The C function takes the first element of the tuple argument, which is the object, and casts it to the CONCRETE type begin defined, and calls the corresponding C++ method, munging arguments as required. [Smly, the return value of the C++ method has to be converted to a PyObject for the C function to return]
There are many ways to do this; using SWIG is one; but it is IRRELEVANT. You can write it by hand for example. The point is that only ONE abstract C++ class is required as a base, and only ONE Python type object is required to wrap it, and this WILL handle every single C++ class suitably derived from the abstract base.
I hope this is clear: the lookup of specific non-virtual method on a class is independent of the _automatically_ polymorphic lookup of the various python virtual methods, including getattr and, as you mentioned, len.
It makes sense, to have a specific derived class, in which the virtual getattr is overriden, and uses a particular data structure and algorithm to do attribute lookup; for example, for the kind of table SWIG or other automatic generator generates.
If I were implementing a getattr method by hand, I'd be tempted to us an STL map, myself, since it is likely to MUCH faster than the standard linear search typically used implementing extension types.
Anyhow, the point is that there is nothing new here, this is no different to C: finding attributes of objects, including methods, has nothing to do with the Python interface to the type object: it is universally achieved by a type specific getattr method.
-- John Skaller, mailto:skaller@maxtal.com.au 10/1 Toxteth Rd Glebe NSW 2037 Australia homepage: http://www.maxtal.com.au/~skaller voice: 61-2-9660-0850
Geoffrey Furnish wrote:
I for one am losing track of the argument here. Everything you say below about the mechanism that Python uses for finding and invoking functions of modules, is well known, and if you didn't read it anywhere first, can be directly observed from the Python C source. The part of your post which relates to a factual rendition of how things works in CPython today, is not in contention.
Good.
Your claim that this can all be handled with a single base class is--until you demonstrate it by posting ocde which works as you posit--at best understood to be an hypothesis.
Fine. It seems like a good hypothesis. I have built some of the code. I know it can work, since I know the Python object model reasonably well, and I know C++ backwards, forwards, and inside out. :-)
Several of us on this list, including myself, Paul, Barry, and doubtless others, have worked out schemes largely similar to what you describe.
What problems were encountered?
I invite you to develop and post code which works as you claim can be done. We would all love to see a tangible incarnation of the design ideas you espouse. In other words, show us the beef.
I have another more pressing project. What I can tell you is that I did indeed begin construction of a standard C++ wrapper, but I didn't bother to complete it because I found a better way to achieve my goals. What I'm saying is that there is a few days work involved, and I won't gain anything out of it, because I'm not that interested in extending CPython using C++ any more. I'm still on this mailing list because I am interested in what others are doing and I'm willing to provide what assistance I can. I can give a sketch of the required architecture. I would be very interested in talking about some of the detail problems, such as the one I mentioned about subtables. But I have no comprehension of how it is not completely obvious what path must be taken. You made a comment something like: 'So now we can call the virtual len, what about the non-virtual methods'. And my point is: that is an implementation detail of the specific getattr hook a particular subclass of the abstract base provides, and those details aren't relevant to writing the C/C++ wrapper. It is quite true, that there are multiple ways to do this. [That is, lookup attributes] But it can be done USING POLYMORPHISM. So it can be separated out, utterly and totally. Exactly as there is more than one way to determine the 'len' of an object. It depends on the derived type, and the derived type provides the answer by overriding the virtual len. In other words, to the point where such specific functions are dynamically loadable. [This is the true test of correct abstraction: when you can dynamically load a class, known ONLY by it's abstract base. Ignorance of any other structure is enforced by the very fact of dynamic loading: it just isn't possible to call a method not in the abstract base, even if you get the run-time type of the object, you don't know the static type.] Consider the 'len' function. Clearly, we need a C++ class like: class Cpp_PyObject { virtual int len() const =0; .. }; How is this called? Well, the python type struct has a pointer in it of type Cpp_PyObject * and there is a single C wrapper function: int len (PyObject *object) { return ((C_Cpp_PyObject *) object)->len() } which is in the 'len' slot in the CPython virtual table. in this example, the C struct C_Cpp_Pyobject contains a pointer to a Cpp_PyObject.
From this example, you can construct all the other required methods.
There are some issues. First, you will note EVERY method is defined, add, radd, etc -- even for C++ types you want to define, that aren't numbers of sequences, etc. This really IS a problem, because CPython checks some things by seeing if the subtable pointer is NULL. So a pragmatic solution might, indeed, requires several abstract bases rather than one -- a base for numbers, another for sequences, etc. [But the number is still finite and fixed for all C++ extensions]. The trick in all this is that _constructors_ are not methods. They're ordinary functions that fill out the Python PyObject struct C_Cpp_PyObject: we construct a client C++ object and store the pointer to it in that struct. BTW: here is another solution, abstract in principle, but technically concrete: class Cpp_PyObject { virtual int len() const { throw "Unimplemented Error"; } .. }; -- John Skaller, mailto:skaller@maxtal.com.au 10/1 Toxteth Rd Glebe NSW 2037 Australia homepage: http://www.maxtal.com.au/~skaller voice: 61-2-9660-0850
Duncan Grisby writes:
I haven't particularly followed the discussion, but I just want to point out that doing what you suggest is extremely dangerous. There is no guarantee that longs are the same size as pointers.
With the notable exception of pointers to members, since when has a long never been large enough to hold a pointer value? Can you name a specific instance where this doesn't work? (or has the C++ standards committee finally decided to break all pointers just for the hell of it?) SWIG does this internally and has never had any problems.
There is also nothing to stop the Python code from modifying the integer, causing the C code to dereference arbitrary memory locations. It shouldn't be possible for Python code to cause core dumps. You should use PyCObjects instead.
Just because you use a CObject doesn't mean you're going to eliminate core dumps. Of course, I personally believe that you should be able to modify the pointer value if you really want to--but that's just me :-). Cheers, Dave
David Beazley writes:
Duncan Grisby writes:
I haven't particularly followed the discussion, but I just want to point out that doing what you suggest is extremely dangerous. There is no guarantee that longs are the same size as pointers.
With the notable exception of pointers to members, since when has a long never been large enough to hold a pointer value? Can you name a specific instance where this doesn't work? (or has the C++ standards committee finally decided to break all pointers just for the hell of it?) SWIG does this internally and has never had any problems.
"Hasn't failed yet" is not the same as "works because it is intended to work". The right way to hold pointer values, is in objects of pointer type. -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski
Geoffrey Furnish writes:
"Hasn't failed yet" is not the same as "works because it is intended to work". The right way to hold pointer values, is in objects of pointer type.
<rant> Yeah, yeah, whatever. For all of the time that C++ facists spend whining about "intended behavior" and the "right" way to do things, I have yet to see a compiler with 13 bit integers, 77 bit longs, and 125 bit pointers, nor do I ever think I will see such a thing in my lifetime. Therefore, semantics aside, there is really nothing wrong with opting for a simple approach if it works and you know what you are doing. Otherwise, I just don't see the point of needlessly making things 10 times more complicated than it has to be. </rant> -- Dave
Geoffrey Furnish writes:
"Hasn't failed yet" is not the same as "works because it is intended to work". The right way to hold pointer values, is in objects of pointer type.
David Beazley <beazley@cs.uchicago.edu>:
<rant> Yeah, yeah, whatever. For all of the time that C++ facists spend whining about "intended behavior" and the "right" way to do things, I have yet to see a compiler with 13 bit integers, 77 bit longs, and 125 bit pointers, nor do I ever think I will see such a thing in my lifetime. Therefore, semantics aside, there is really nothing wrong with opting for a simple approach if it works and you know what you are doing. Otherwise, I just don't see the point of needlessly making things 10 times more complicated than it has to be. </rant>
Rant aside, what compiler is going to support different datatype sizes for stuff declared extern "C" than for native C++ types? Get real. --Guido van Rossum (home page: http://www.python.org/~guido/)
O
Rant aside, what compiler is going to support different datatype sizes for stuff declared extern "C" than for native C++ types? Get real.
--Guido van Rossum (home page: http://www.python.org/~guido/)
_______________________________________________ C++-SIG maillist - C++-SIG@python.org http://www.python.org/mailman/listinfo/c++-sig
Gotcha. IBM, that's who. It was IBM who insisted on the C++ standard specifiying that a pointer to an extern "C" function and a pointer to a C++ function with the same signature are type incompatible. Without this provision they claimed they could not implement C++ on one of their mainframes. I don't have the article in front of me but this point was the subject of a column in the C++ Report (Aug 99, I think). I have some improvements to CXX thanks to Barry Scott but I am going to have to figure out what this annoying change to the standard means before doing any more.
Gotcha.
IBM, that's who. It was IBM who insisted on the C++ standard specifiying that a pointer to an extern "C" function and a pointer to a C++ function with the same signature are type incompatible. Without this provision they claimed they could not implement C++ on one of their mainframes. I don't have the article in front of me but this point was the subject of a column in the C++ Report (Aug 99, I think).
Aiiiigh!!! Make it stop! Make it stop! First off, I want to apologize for my out-of-line facist comment from earlier--having spent far too much time trying to sort out changes in the C++ spec recently has turned this into a hot-button for me (so much so, I was about 30 seconds away from deleting all signs of C++ from my machines last month). As for the casting to long issue, Geoffrey pointed out to me privately that you can't do this if you care about inheritance which is, of course, completely correct. SWIG, although it does cast to a long at some point in its internal pointer handling, also keeps track of type information for this purpose. For instance, depending on the type, it is necessary to perform a casting operation to correctly adjust the value of a pointer when cast from a derived class to a base class. If you're extremely lucky, you can avoid this step and things will work, but there's no guarantee that it will work everywhere and it definitely won't work with multiple inheritance. However, with that said, I'm still a big fan of keeping things as simple as is reasonable. As for the extern "C" issue, I would love to know how to officially resolve that issue. For now, I've solved it by simply slapping an extern "C" around everything that gets hooked to Python (all wrapper functions). Still, it doesn't seem like a very satisfactory solution. I have no idea what it implies for something like CXX. Cheers, Dave
David Beazley wrote:
Aiiiigh!!! Make it stop! Make it stop!
Try ocaml :-)
However, with that said, I'm still a big fan of keeping things as simple as is reasonable.
As for the extern "C" issue, I would love to know how to officially resolve that issue.
First you read the Standard. If you can't work out what it means, you ask a committee delegate (such as me). If I can't figure it out, which is likely, I can post the question directly on the committee email reflector, and echo the responses back to you.
For now, I've solved it by simply slapping an extern "C" around everything that gets hooked to Python (all wrapper functions). Still, it doesn't seem like a very satisfactory solution. I have no idea what it implies for something like CXX.
You must assume your client (possibly you) has a C/C++ compiler pair, and Python was built with the C part of it (except for main, which must be built with C++ to ensure the run time is set up for exception handling and RTTI). When it comes to DLLS/shared libraries, you must rely on vendor specs, and luck. The same applies to threading, and calling ANY system function other than those required by the Standard. That is, basically, Guido's "get real" approach is pretty close to the right idea. :-) -- John Skaller, mailto:skaller@maxtal.com.au 10/1 Toxteth Rd Glebe NSW 2037 Australia homepage: http://www.maxtal.com.au/~skaller voice: 61-2-9660-0850
For the list: The most contentious aspects of this tiff are being handled in private email. However I would like to comment publicly on one aspect of Guido's post: Guido van Rossum writes:
Rant aside, what compiler is going to support different datatype sizes for stuff declared extern "C" than for native C++ types? Get real.
One of my personal beefs with the casting to PyCFunction, has nothing to do with the (illegal according to strict C++ rules) presumptions about function pointer compatibility. There is also the issue of just exactly what the function signature is. The Python C implementation actually casts all function pointers to PyCFunction, even if they have different arg lists! The fact that this is illegal by C++ language rules is almost the least of all concerns! The size of the pointer type is almost irrelevant. Here we have type information (the function invocation arg list) being communicated through independent parameters. Okay, fine, there aren't a lot of choices in C. Whatever. In C++ though, the compiler actually supports a decent type system, and there really is no reason to attempt to subvert the C++ type system and reincarnate its soul in the form of extraneous parameters, whose definitions have to be predetermined and extensions to which cannot be properly anticipated by the implementor. These sorts of problems can be solved in C++ by letting the compiler's type system do the work for you. For this case in question, probably its going to have to mean some template trampolines and all the associated machinery. And unfortunately, this reliance upon templates has caused a lot of fuss on this SIG. Frnakly, I don't want to get into a C versus C++ style debate. I really couldn't care less how people write their C code. Observe that I have never posted to c.l.p suggesting that "Python's implementation would be a lot nicer if it was swtiched from C to C++". That does happen to be my personal oppinion, but I don't advocate it publicly because such advocacy isn't going to change the situation, and just creates needless stir and resentment. But by the same token, I really don't think it's fair for people with an ax to grind against C++ to use the C++ SIG to vent about how dogmatic the C++ camp is. Seriously, couldn't we all just calm down and work on solving the problem? -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski
Guido van Rossum wrote:
Rant aside, what compiler is going to support different datatype sizes for stuff declared extern "C" than for native C++ types? Get real.
Actually, there is a kind of understanding on the committee of the concept of 'compatible C and C++ compilers' aka C/C++ compiler. This arose because 1) many compilers are C/C++ compilers: same compiler, with a mode switch. 2) there is considerable difficulty expressing any requirements on what 'extern "C"' actually means, otherwise. Also there is some reason for there to be a difference between C and C++ on some systems, especially in the case of things like function pointers. But, there is another issue here: for function pointers, what exactly does 'extern "C"' mean if the _parameters_ are, for example, an enum. Or another function pointer?? In my opinion, it is reasonable to target a product like Python on a C/C++ compiler pair. It is a pragmatic choice, excluding few platforms/systems (and even in these cases, patches may provide a fix). Just be wary of 'get reals': from being on the committee I know there are a LOT of examples where people aren't doing quite what you'd expect, claim to have a good reason for it, and insist their implementation not be broken by a gratuitous rule :-) Sigh. A final explanation. The ISO C++ Standard does NOT define any C++ language. It defines 'conforming translator'. [This is not the case for C]. Many different languages can be compiled by a conforming translator, the specification is said to be parameterized by certain rules and quantities which the vendor is required to provide: i.e. a 'C++ compiler' is NOT a piece of software. It is a piece of software AND some documentation. This is an excellent device, because it allows a software vendor to say: 'this software requires a C++ compiler satisfying the following additional constraints', and it is up to the client to find such a product to build the software. -- John Skaller, mailto:skaller@maxtal.com.au 10/1 Toxteth Rd Glebe NSW 2037 Australia homepage: http://www.maxtal.com.au/~skaller voice: 61-2-9660-0850
All I can say is, the more I hear about C++, the less likely it becomes that Python 2.0 will be written in it. --Guido van Rossum (home page: http://www.python.org/~guido/)
Ephi Dror wrote:
Does anyone know how much Python is really used on Redhat Linux with the conjunction with C++, especially the direction of Python calling C++ functions?
We are developing version 2 of a large application that will be centered around Python with several large, complex C++ extensions. The application is targeted at embedded software developers and provides visibility into their embedded target via source code instrumentation, and analysis of the resulting "tag stream" as their target runs. The product is called CodeTEST (www.amc.com). We have been shipping a "reduced" version of this for a year now that allows customers to script tests for source code coverage with great success. The only reason I am providing this information is to add some context, and hopefully, some believability to the claims that will follow. The only difference is that we ship a python we build with the product, as we cannot depend upon our customers to already have it installed. We ship on Windows NT, Solaris, and soon Linux. So, the answer to your question is yes, we use Python to supply an interface to our C++ libraries. To date it works quite well on all platforms that we develop on. This is not to say there have been no difficulties. But, they have not been insurmountable, and the payoff for using Python is too great to let a few annoyances impede progress. Besides, to date most of our problems have been centered around C++ itself. Our C++ components make heavy use of STL, and getting them to work with both VC++ and g++ can be a challenge. Mostly due to the inadequate STL shipped with VC++ 6.0.
We have not given up yet but we are close. I would love to hear from the group that I am wrong. That I was just blind and simply did not get it. That calling C++ class from Python is really a piece of cake after all and especially on RedHat 6.1 Linux with standard installation components.
I believe you are wrong. Calling a C++ class is not too difficult. I will take an extension I wrote recently, and pair it down to something very simple. I think the thing you missed in your simple example is not creating a new type. Trying to pass C++ object pointers around as other things is not the best road to travel in my opinion. If you extend Python with a new type, you can do type checking on the self argument, and make your extension more robust in the presence of other programmers. I will mail the sample extension to this list in a few hours.
Thank you all for helping us.
We hope to stay in the group and contribute in the future, but it all depend on what next for us in terms of using Python/C++.
Ephi.
-Ian Searle Engineering Manager Software Analysis Tools Applied Microsystems Corporation http://www.amc.com
Does anyone know how much Python is really used on Redhat Linux with the conjunction with C++, especially the direction of Python calling C++ functions?
It works wonderfully. Using SWIG even makes it simple. See http://alldunn.com/wxPython/ for an example. -- Robin Dunn Software Craftsman robin@AllDunn.com http://AllDunn.com/robin/ http://AllDunn.com/wxPython/ Check it out!
Ephi Dror writes:
Unfortunately, as of this moment, we have not yet successfully run our simple spam program.
I'd like to respond to some parts of your message. I will confess up front that I have not analyzed your recent posts personally in any detail whatsoever, and have not followed the technical exchange. I'm wading in an aligator pond, fighting for my life right now in my day job, and simply couldn't afford the distraction. I've gotten the vague impression that you were given good advice, but can easily believe you haven't yet found all the necessary tricks of the trade. So, with that background, I'd like to comment on the non-technical aspects of your post.
It is partially worked but then we got the two unexplained strange phenomena that as of now, nobody seems to understand why. Also, even if we got those problems solved using compiler xyz version 123 who knows what other problems are waiting for us in the next corner.
This is an important reason why, imo, the focus of this sig should be on constructing facilities that are correct according to a strict reading of the ISO C++ language definition. Compiler implementation quality variations will always be a fact of life. But there is an important qualitative distinction between unsupported features, and unsupportable features. It may be desirable in the eyes of many to limit use of the former, but it is /imperative/ to avoid use of the later.
I am personally very disappointed from all of this. I really truly wanted to give python a chance. I got the impression by reading one of the Python books back cover that what we want to do is common practice and should be a plug and play . (http://www.amazon.com/exec/obidos/ts/book-reviews/1565921976/103-5409830-171...) Reading the books give the impression that Python and C++ were meant to live in peace and be haply working with each other. Unfortunately, my personal experience realty proved the opposite.
Well, there are shades of truth on both sides. C code is C++ code, so of course, techniques which work with Python/C, will work with Python/C++ too. If you just stick to the part of C++ which happens to be equivalent to C, then you can certainly get the examples from the books to work basically without change. But if you do that, you aren't really "using C++". And therein lies the rub.
Python may work nicely with C code, but who needs that 30 days before the year 2000 !!, Programmers today need at least integration with existing C++ code and very soon they need integration with Java code. Our example only had the simplest C++ class in the word, and Python could not even use it correctly. Is just un believable to me. It is possible that we made stupid mistakes in the code, it is also possible that we are bad programmers. Who knows?. We did not hide our code. In fact, I attached it to my original email.
Well, like I said above, I haven't analyzed your posts for technical details, so I can't say whether you made avoidable mistakes. But I definitely do agree with you that it is roughly 100 times harder than it needs to be, to use Python from C++. The charter of this sig is to fix that, by developing the additional software that would allow Python's compiled interface to be exploited from C++ "with ease". The first and most basic issue, is compiling Python so it initializes C++ global objects correctly. There is a patch on the sig's www site to help with that. Beyond that, it should be possible to call C++ functions, both global functions and member functions, easily. Several of us have efforts in this area under way, but no consensus has really emerged yet. Beyond that, it would be desirable to abolish the horid NULL checking that is so endemic to Python C extension authorship. PyExceptions should be real C++ exceptions. Beyond that, you'd like to have a C++ friendly API for Python services. Beyond that, you'd like to have classes to encapsulate the API functional subsystems in the Python C API. At that point, programming Python from C++ would be "easy". We're not there yet.
Does anyone know how much Python is really used on Redhat Linux with the conjunction with C++, especially the direction of Python calling C++ functions?
We have not given up yet but we are close. I would love to hear from the group that I am wrong.
I don't think you're wrong. But I do think you may be missing the purpose of the SIG. The SIG isn't really here for user support--at least that's not the primary mission. The primary mission of this SIG, at this point in time, is to FIX THE PROBLEM!!! Python is unnecessarily hard to extend. That is not to say it is iimpossible, or even that it is too hard to be worth it on an absolute scale. Ian gave a detailed description of one effort. I have personal knowledge of extensive extension effort underway in the DOE labs, at NASA, and beyond, and these efforts have been featured at length in public talks at Python workshops over the years. There are doubtless hundreds more such development projects at institutions all over the world. So it can be done. But it could be done much more easily, if the right supporting infrastructure is developed. At this point in time, that is the primary mission of this sig. At some future date, after we have reached a point where the SIG has something tangible and generally agreed upon, to put forward to the larger community for use, and hopefully ultimately for direct incorporation into the Python core, then the focus of this SIG will change to more of a user support organization. But we have an immense amount of work to do before we get to that point.
That I was just blind and simply did not get it. That calling C++ class from Python is really a piece of cake after all and especially on RedHat 6.1 Linux with standard installation components.
I would encourage you to view your recent experiences not as a cause for dismissal of Python as a programming environment, but as a personal challenge for involvement to improve the situation for the common good. Python is community devleopment project, just as most free software is. Someone was recently inquring to know when GCC was ever going to be upgraded to suppor ISO C++, and someone (Nathan Myers?) wrote back "Sooner if you help". So. When will Python be easy to use from C++? Sonner if you help.
Thank you all for helping us.
We hope to stay in the group and contribute in the future, but it all depend on what next for us in terms of using Python/C++.
Grab a keyboard, and get to work! Write something that fixes some part of the problems you've encountered, and lets get it posted for public comment. -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski
On Wednesday 1 December, Geoffrey Furnish wrote:
The first and most basic issue, is compiling Python so it initializes C++ global objects correctly. There is a patch on the sig's www site to help with that.
Unfortunately, I don't believe that that patch addresses the full problem with global initialisers. It guarantees that global initialisers will be called for object files linked with the actual Python executable, since the main() function is C++. Unfortunately, compiling main() with C++ gives no guarantees about initialisation of shared libraries dynamically loaded at run time. It is my understanding that a sizeable portion of C / C++ code called from Python is accessed as extension modules loaded at run-time. The code I've written certainly is. C++ as a language gives no guarantees about how and whether static initialisers in extension modules are handled. I don't think Python can do much to help with this -- it's fundamentally a linker and operating system issue. Fortunately, it seems that the majority of platforms do handle this correctly. All ELF-based Unixes should do. Windows NT also does. The only platform with a problem I have encountered so far is AIX, and that can be fixed with a small patch to Python which uses an AIX specific call rather than libdl. I'm not saying that the patch to compile main() with C++ is wrong -- it's just not sufficient. Cheers, Duncan. -- -- Duncan Grisby \ Research Engineer -- -- AT&T Laboratories Cambridge -- -- http://www.uk.research.att.com/~dpg1 --
participants (12)
-
Adrian Eyre -
David Beazley -
Dror, Ephi -
Duncan Grisby -
Ephi Dror -
Geoffrey Furnish -
Guido van Rossum -
Ian Searle -
Kevin Dahlhausen -
Paul F. Dubois -
Robin Dunn -
skaller