Ephi, I'd suggest you take a look at SWIG - the simplified wrapper interface generator. I've had great success in using it to wrap the C++ cross-platform GUI FLTK. If you need static constructors to run, I think you can use the %init statment somehow, even with a C-based Python interpreter. Python is all people say it is. The only thing I don't like about it compared to straight C++ is that it is not easy to provide users with a single run-time executable - but then they should have Python installed anyway! :)

Reading the books give the impression that Python and C++ were meant to live in peace and be haply working with each other.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Sr Software Engineer, Morse Controls http://www.geocities.com/SiliconValley/Peaks/4673 "Do or do not. There is no 'Try.' " -Yoda

On Wednesday 1 December, "Adrian Eyre" wrote: [...]

'Spam' is a C++ object, not a python object, and you can't just cast it to a PyObject* and expect it to work. One way to do this is to pass back a pointer to the object as a long:

static PyObject* ext_new_Spam(PyObject* self, PyObject* args) { int a, b;

if (!PyArg_ParseTuple(args, "ii", &a, &b)) return NULL;

Spam* spam = new Spam(a, b); return Py_BuildValue("l", (long) spam); // Or return PyInt_FromLong((long)

I haven't particularly followed the discussion, but I just want to point out that doing what you suggest is extremely dangerous. There is no guarantee that longs are the same size as pointers. There is also nothing to stop the Python code from modifying the integer, causing the C code to dereference arbitrary memory locations. It shouldn't be possible for Python code to cause core dumps. You should use PyCObjects instead. Cheers, Duncan. -- -- Duncan Grisby \ Research Engineer -- -- AT&T Laboratories Cambridge -- -- http://www.uk.research.att.com/~dpg1 --

There's no guarantee of most types in C++, only that a long is at least 1 byte.

From K & R, 2nd ed., 2:2, verse 8, p. 36:

"Each compiler is free to choose appropriate sizes for its own hardware, subject only to the restriction that shorts and ints are at least 16 bits, longs are at least 32 bits, and short is no longer than int, which is no longer than long." Everyone may be seated now... Cheers, Dave

Adrian Eyre writes:

That's rather annoying. What if my processor only has 1-bit buses, and my RAM only stores 10 words. I could never have any integers. :(

What, you mean like if you're working on air-traffic control? :-). -- Dave

skaller writes:

David Beazley wrote: The conversion of pointers to ints is not supported in C++:

Right. Pointer values must be held in objects of pointer type. And anticipating the following, I'll go ahead and state that pointer values must be held in objects of pointer type which are of the same type as the pointer in question (or which are accessible through valid pointer conversions).

For those using C Python sources with C++, there are two gotchas.

1) Enumerations are NOT integers in C++ [Luckily, Guido doesn't use them much]

Just to amplify, you can always make an int from an enum if you need to. A common case where this issue comes up, is in situations like the following: enum { This, That }; vector<int>::iterator it = find( v.begin(), v.end(), That ); which will not work (will not compile), but which can be repaired by doing: vector<int>::iterator it = find( v.begin(), v.end(), int(That) );

2) The very core of Python contains a serious breach of C++ rules: it casts functions pointers, which is NOT allowed AT ALL.

[I'm responsible for that rule!]

Thank you!

Luckily, this actually works on most C++ compilers.

I disagree with this. It does not work on /any/ "C++" compilers. There are however, on the market, many compilers which support any of a family of as yet undocumented languages, which seem all to be similar to "C++" in many ways, but which evidently support various "extension" constructs that are illegal in "C++", such as casting pointers. The right thing to do, is to avoid efforts to cast function pointers. For the Python/C++ interface, this basically means you need a seperate extension type object for each C++ object you wish to register as a Python extension. Making this machinery work easily and maintainably seems to be challenging. I don't personally have much hope for doing it, except through heavy use of template mechanisms, which have been the source of much consternation on this sig. -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski

skaller writes:

Geoffrey Furnish wrote:

...

For the Python/C++ interface, this basically means you need a seperate extension type object for each C++ object you wish to register as a Python extension.

I don't believe this is the case. See below. [...] I will give an example, by taking a tiny subset of the python virtual functions: the len function, for example:

class cpp_py_object { virtual int len() const = 0; ... };

Now, by using THIS class as the sole base of all cpp extension types, you only need a single CPython type, with a pointer to the C++ object of type cpp_py_object.

Okay, great. So we can invoke the virtual function len() on an extension object of unknown type. Now what's the plan for invoking non virtual methods of C++ objects without knowing their type? -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski

I for one am losing track of the argument here. Everything you say below about the mechanism that Python uses for finding and invoking functions of modules, is well known, and if you didn't read it anywhere first, can be directly observed from the Python C source. The part of your post which relates to a factual rendition of how things works in CPython today, is not in contention. Your claim that this can all be handled with a single base class is--until you demonstrate it by posting ocde which works as you posit--at best understood to be an hypothesis. Several of us on this list, including myself, Paul, Barry, and doubtless others, have worked out schemes largely similar to what you describe. What has not yet been done, is to field an implementation that both has the desired features, works with exactly one base class for all extensions, and is correct according to the C++ language standard. I have pursued a slightly different approach which is correct according to the language standard, but uses templates to fabricate on demand, much of the machinery which is neded to make all the funciton invocations. I am sorry that I have not yet been able to get this fully dusted off and whipped into presentable form. Still working on that. Meanwhile, Paul is pursing a system which does not have that characteristic, but instead works more like what you describe. But he is also not done, and still working out final details. I invite you to develop and post code which works as you claim can be done. We would all love to see a tangible incarnation of the design ideas you espouse. In other words, show us the beef. skaller writes:

Geoffrey Furnish wrote:

...

Okay, great. So we can invoke the virtual function len() on an extension object of unknown type.

yes.

...

Now what's the plan for invoking non virtual methods of C++ objects without knowing their type?

That is impossible. It is not necessary to be able to do this, since the Python parser cannot generate any code which calls it. Bytecode can only call 'virtual functions' on PyObjects.

Perhaps, you mean, how do we do this:

object.method(arguments)

And the answer is: the same way python does. It is the only way. Python calls 'getattr' to fetch 'method' by name. The resulting function fetched has a fixed calling sequence. Otherwise the bytecode interpreter could never call it. It accepts a pair of arguments, a tuple and a dictionary.

[Actually, there is a complication with an obsoleted flag, determining if **kwds is support -- but lets ignore that for clarity].

In other words, it is impossible to call ANYTHING from python OTHER than one of the virtual methods. And so that is all that needs to be provided.

Now, this is not to say that C++ cannot call other C++ method directly. But you have to understand that a C++ method like

void method (int x, float y) const

cannot ever be called by Python. Here's the sequence of events:

1) Python calls the C 'getattr' method of the single, fixed C++ wrapper type: this is a C function.

2) The C function casts the object pointer to the abstract base type of the C++ heirarchy.

3) Then, it calls the C++ virtual getattr method, after suitably rearranging the arguments.

4) The C++ getattr method has to return something that can be converted into a function which Python can call; that is, it has to return a C function which takes a tuple and dictionary as an argument.

It finds this function in a table. Because C is a bit brain dead, there is no way to create a closure here. [This is possible using Viper, since the implementation language, ocaml, supports them: generic extensions in Viper are much easier than in C or C++ [but binding existing C libraries is, conversely, harder]

The C function takes the first element of the tuple argument, which is the object, and casts it to the CONCRETE type begin defined, and calls the corresponding C++ method, munging arguments as required. [Smly, the return value of the C++ method has to be converted to a PyObject for the C function to return]

There are many ways to do this; using SWIG is one; but it is IRRELEVANT. You can write it by hand for example. The point is that only ONE abstract C++ class is required as a base, and only ONE Python type object is required to wrap it, and this WILL handle every single C++ class suitably derived from the abstract base.

I hope this is clear: the lookup of specific non-virtual method on a class is independent of the _automatically_ polymorphic lookup of the various python virtual methods, including getattr and, as you mentioned, len.

It makes sense, to have a specific derived class, in which the virtual getattr is overriden, and uses a particular data structure and algorithm to do attribute lookup; for example, for the kind of table SWIG or other automatic generator generates.

If I were implementing a getattr method by hand, I'd be tempted to us an STL map, myself, since it is likely to MUCH faster than the standard linear search typically used implementing extension types.

Anyhow, the point is that there is nothing new here, this is no different to C: finding attributes of objects, including methods, has nothing to do with the Python interface to the type object: it is universally achieved by a type specific getattr method.

-- John Skaller, mailto:skaller@maxtal.com.au 10/1 Toxteth Rd Glebe NSW 2037 Australia homepage: http://www.maxtal.com.au/~skaller voice: 61-2-9660-0850

Duncan Grisby writes:

I haven't particularly followed the discussion, but I just want to point out that doing what you suggest is extremely dangerous. There is no guarantee that longs are the same size as pointers.

With the notable exception of pointers to members, since when has a long never been large enough to hold a pointer value? Can you name a specific instance where this doesn't work? (or has the C++ standards committee finally decided to break all pointers just for the hell of it?) SWIG does this internally and has never had any problems.

There is also nothing to stop the Python code from modifying the integer, causing the C code to dereference arbitrary memory locations. It shouldn't be possible for Python code to cause core dumps. You should use PyCObjects instead.

Just because you use a CObject doesn't mean you're going to eliminate core dumps. Of course, I personally believe that you should be able to modify the pointer value if you really want to--but that's just me :-). Cheers, Dave

Geoffrey Furnish writes:

"Hasn't failed yet" is not the same as "works because it is intended to work". The right way to hold pointer values, is in objects of pointer type.

<rant> Yeah, yeah, whatever. For all of the time that C++ facists spend whining about "intended behavior" and the "right" way to do things, I have yet to see a compiler with 13 bit integers, 77 bit longs, and 125 bit pointers, nor do I ever think I will see such a thing in my lifetime. Therefore, semantics aside, there is really nothing wrong with opting for a simple approach if it works and you know what you are doing. Otherwise, I just don't see the point of needlessly making things 10 times more complicated than it has to be. </rant> -- Dave

O

Rant aside, what compiler is going to support different datatype sizes for stuff declared extern "C" than for native C++ types? Get real.

--Guido van Rossum (home page: http://www.python.org/~guido/)

_______________________________________________ C++-SIG maillist - C++-SIG@python.org http://www.python.org/mailman/listinfo/c++-sig

Gotcha. IBM, that's who. It was IBM who insisted on the C++ standard specifiying that a pointer to an extern "C" function and a pointer to a C++ function with the same signature are type incompatible. Without this provision they claimed they could not implement C++ on one of their mainframes. I don't have the article in front of me but this point was the subject of a column in the C++ Report (Aug 99, I think). I have some improvements to CXX thanks to Barry Scott but I am going to have to figure out what this annoying change to the standard means before doing any more.

David Beazley wrote:

Aiiiigh!!! Make it stop! Make it stop!

Try ocaml :-)

However, with that said, I'm still a big fan of keeping things as simple as is reasonable.

As for the extern "C" issue, I would love to know how to officially resolve that issue.

First you read the Standard. If you can't work out what it means, you ask a committee delegate (such as me). If I can't figure it out, which is likely, I can post the question directly on the committee email reflector, and echo the responses back to you.

For now, I've solved it by simply slapping an extern "C" around everything that gets hooked to Python (all wrapper functions). Still, it doesn't seem like a very satisfactory solution. I have no idea what it implies for something like CXX.

You must assume your client (possibly you) has a C/C++ compiler pair, and Python was built with the C part of it (except for main, which must be built with C++ to ensure the run time is set up for exception handling and RTTI). When it comes to DLLS/shared libraries, you must rely on vendor specs, and luck. The same applies to threading, and calling ANY system function other than those required by the Standard. That is, basically, Guido's "get real" approach is pretty close to the right idea. :-) -- John Skaller, mailto:skaller@maxtal.com.au 10/1 Toxteth Rd Glebe NSW 2037 Australia homepage: http://www.maxtal.com.au/~skaller voice: 61-2-9660-0850

Guido van Rossum wrote:

Rant aside, what compiler is going to support different datatype sizes for stuff declared extern "C" than for native C++ types? Get real.

Actually, there is a kind of understanding on the committee of the concept of 'compatible C and C++ compilers' aka C/C++ compiler. This arose because 1) many compilers are C/C++ compilers: same compiler, with a mode switch. 2) there is considerable difficulty expressing any requirements on what 'extern "C"' actually means, otherwise. Also there is some reason for there to be a difference between C and C++ on some systems, especially in the case of things like function pointers. But, there is another issue here: for function pointers, what exactly does 'extern "C"' mean if the _parameters_ are, for example, an enum. Or another function pointer?? In my opinion, it is reasonable to target a product like Python on a C/C++ compiler pair. It is a pragmatic choice, excluding few platforms/systems (and even in these cases, patches may provide a fix). Just be wary of 'get reals': from being on the committee I know there are a LOT of examples where people aren't doing quite what you'd expect, claim to have a good reason for it, and insist their implementation not be broken by a gratuitous rule :-) Sigh. A final explanation. The ISO C++ Standard does NOT define any C++ language. It defines 'conforming translator'. [This is not the case for C]. Many different languages can be compiled by a conforming translator, the specification is said to be parameterized by certain rules and quantities which the vendor is required to provide: i.e. a 'C++ compiler' is NOT a piece of software. It is a piece of software AND some documentation. This is an excellent device, because it allows a software vendor to say: 'this software requires a C++ compiler satisfying the following additional constraints', and it is up to the client to find such a product to build the software. -- John Skaller, mailto:skaller@maxtal.com.au 10/1 Toxteth Rd Glebe NSW 2037 Australia homepage: http://www.maxtal.com.au/~skaller voice: 61-2-9660-0850

Ephi Dror wrote:

Does anyone know how much Python is really used on Redhat Linux with the conjunction with C++, especially the direction of Python calling C++ functions?

We are developing version 2 of a large application that will be centered around Python with several large, complex C++ extensions. The application is targeted at embedded software developers and provides visibility into their embedded target via source code instrumentation, and analysis of the resulting "tag stream" as their target runs. The product is called CodeTEST (www.amc.com). We have been shipping a "reduced" version of this for a year now that allows customers to script tests for source code coverage with great success. The only reason I am providing this information is to add some context, and hopefully, some believability to the claims that will follow. The only difference is that we ship a python we build with the product, as we cannot depend upon our customers to already have it installed. We ship on Windows NT, Solaris, and soon Linux. So, the answer to your question is yes, we use Python to supply an interface to our C++ libraries. To date it works quite well on all platforms that we develop on. This is not to say there have been no difficulties. But, they have not been insurmountable, and the payoff for using Python is too great to let a few annoyances impede progress. Besides, to date most of our problems have been centered around C++ itself. Our C++ components make heavy use of STL, and getting them to work with both VC++ and g++ can be a challenge. Mostly due to the inadequate STL shipped with VC++ 6.0.

We have not given up yet but we are close. I would love to hear from the group that I am wrong. That I was just blind and simply did not get it. That calling C++ class from Python is really a piece of cake after all and especially on RedHat 6.1 Linux with standard installation components.

I believe you are wrong. Calling a C++ class is not too difficult. I will take an extension I wrote recently, and pair it down to something very simple. I think the thing you missed in your simple example is not creating a new type. Trying to pass C++ object pointers around as other things is not the best road to travel in my opinion. If you extend Python with a new type, you can do type checking on the self argument, and make your extension more robust in the presence of other programmers. I will mail the sample extension to this list in a few hours.

Thank you all for helping us.

We hope to stay in the group and contribute in the future, but it all depend on what next for us in terms of using Python/C++.

Ephi.

-Ian Searle Engineering Manager Software Analysis Tools Applied Microsystems Corporation http://www.amc.com

Ephi Dror writes:

Unfortunately, as of this moment, we have not yet successfully run our simple spam program.

I'd like to respond to some parts of your message. I will confess up front that I have not analyzed your recent posts personally in any detail whatsoever, and have not followed the technical exchange. I'm wading in an aligator pond, fighting for my life right now in my day job, and simply couldn't afford the distraction. I've gotten the vague impression that you were given good advice, but can easily believe you haven't yet found all the necessary tricks of the trade. So, with that background, I'd like to comment on the non-technical aspects of your post.

It is partially worked but then we got the two unexplained strange phenomena that as of now, nobody seems to understand why. Also, even if we got those problems solved using compiler xyz version 123 who knows what other problems are waiting for us in the next corner.

This is an important reason why, imo, the focus of this sig should be on constructing facilities that are correct according to a strict reading of the ISO C++ language definition. Compiler implementation quality variations will always be a fact of life. But there is an important qualitative distinction between unsupported features, and unsupportable features. It may be desirable in the eyes of many to limit use of the former, but it is /imperative/ to avoid use of the later.

I am personally very disappointed from all of this. I really truly wanted to give python a chance. I got the impression by reading one of the Python books back cover that what we want to do is common practice and should be a plug and play . (http://www.amazon.com/exec/obidos/ts/book-reviews/1565921976/103-5409830-171...) Reading the books give the impression that Python and C++ were meant to live in peace and be haply working with each other. Unfortunately, my personal experience realty proved the opposite.

Well, there are shades of truth on both sides. C code is C++ code, so of course, techniques which work with Python/C, will work with Python/C++ too. If you just stick to the part of C++ which happens to be equivalent to C, then you can certainly get the examples from the books to work basically without change. But if you do that, you aren't really "using C++". And therein lies the rub.

Python may work nicely with C code, but who needs that 30 days before the year 2000 !!, Programmers today need at least integration with existing C++ code and very soon they need integration with Java code. Our example only had the simplest C++ class in the word, and Python could not even use it correctly. Is just un believable to me. It is possible that we made stupid mistakes in the code, it is also possible that we are bad programmers. Who knows?. We did not hide our code. In fact, I attached it to my original email.

Well, like I said above, I haven't analyzed your posts for technical details, so I can't say whether you made avoidable mistakes. But I definitely do agree with you that it is roughly 100 times harder than it needs to be, to use Python from C++. The charter of this sig is to fix that, by developing the additional software that would allow Python's compiled interface to be exploited from C++ "with ease". The first and most basic issue, is compiling Python so it initializes C++ global objects correctly. There is a patch on the sig's www site to help with that. Beyond that, it should be possible to call C++ functions, both global functions and member functions, easily. Several of us have efforts in this area under way, but no consensus has really emerged yet. Beyond that, it would be desirable to abolish the horid NULL checking that is so endemic to Python C extension authorship. PyExceptions should be real C++ exceptions. Beyond that, you'd like to have a C++ friendly API for Python services. Beyond that, you'd like to have classes to encapsulate the API functional subsystems in the Python C API. At that point, programming Python from C++ would be "easy". We're not there yet.

Does anyone know how much Python is really used on Redhat Linux with the conjunction with C++, especially the direction of Python calling C++ functions?

We have not given up yet but we are close. I would love to hear from the group that I am wrong.

I don't think you're wrong. But I do think you may be missing the purpose of the SIG. The SIG isn't really here for user support--at least that's not the primary mission. The primary mission of this SIG, at this point in time, is to FIX THE PROBLEM!!! Python is unnecessarily hard to extend. That is not to say it is iimpossible, or even that it is too hard to be worth it on an absolute scale. Ian gave a detailed description of one effort. I have personal knowledge of extensive extension effort underway in the DOE labs, at NASA, and beyond, and these efforts have been featured at length in public talks at Python workshops over the years. There are doubtless hundreds more such development projects at institutions all over the world. So it can be done. But it could be done much more easily, if the right supporting infrastructure is developed. At this point in time, that is the primary mission of this sig. At some future date, after we have reached a point where the SIG has something tangible and generally agreed upon, to put forward to the larger community for use, and hopefully ultimately for direct incorporation into the Python core, then the focus of this SIG will change to more of a user support organization. But we have an immense amount of work to do before we get to that point.

That I was just blind and simply did not get it. That calling C++ class from Python is really a piece of cake after all and especially on RedHat 6.1 Linux with standard installation components.

I would encourage you to view your recent experiences not as a cause for dismissal of Python as a programming environment, but as a personal challenge for involvement to improve the situation for the common good. Python is community devleopment project, just as most free software is. Someone was recently inquring to know when GCC was ever going to be upgraded to suppor ISO C++, and someone (Nathan Myers?) wrote back "Sooner if you help". So. When will Python be easy to use from C++? Sonner if you help.

Thank you all for helping us.

We hope to stay in the group and contribute in the future, but it all depend on what next for us in terms of using Python/C++.

Grab a keyboard, and get to work! Write something that fixes some part of the problems you've encountered, and lets get it posted for public comment. -- Geoffrey Furnish Actel Corporation furnish@actel.com Senior Staff Engineer 955 East Arques Ave voice: 408-522-7528 Placement & Routing Sunnyvale, CA 94086-4533 fax: 408-328-2303 "... because only those who write the code truly control the project." -- Jamie Zawinski