Mailman 3 May 2000 - Cplusplus-sig

Trouble getting a method called
by Steve Harris May 22, 2000

May 22, 2000

I'm confused about the interaction between add_varargs_method() and getattr() in PythonExtension<>-derived types. First, I tried defining a method, registering it with add_varargs_method() in my init_type() function, then calling the method from Python. My aim is to be as minimal as possible. Apparently my type isn't exposing the method properly: > python Python 1.5.2 (#3, May 22 2000, 11:01:12) [GCC 2.95.2 19991024 (release)] on sunos5 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> from seh_test import * >>> p = some_pod( "a string" ) Constructing some_pod( "a string" ): 0xd8c60 >>> p.do_it() Traceback (innermost last): File "<stdin>", line 1, in ? AttributeError: 'some_pod' object has no attribute 'do_it' >>> ^D Destructing some_pod: 0xd8c6 At that point, I realized that perhaps methods are just special functors returned from getattr(), so I tried to complete that implementation as follows: ================================================== Py::Object py_bound_pod::do_it(const Py::Tuple& t) { // ... t.verify_length( 0 ); m_pod.do_it(); return Py::Nothing(); } // .... Py::Object py_bound_pod::getattr(const char* pcsz) { return getattr_methods( pcsz ); } void py_bound_pod::init_type() { Py::PythonType& pytype = behaviors(); pytype.name( "some_pod" ); pytype.doc( "A test POD type constructed from a string" ); pytype.supportRepr(); pytype.supportGetattr(); add_varargs_method( "do_it", &py_bound_pod::do_it, "dump out the nested string" ); } ================================================== With this setup, I get a different error: >>> p.do_it() Traceback (innermost last): File "<stdin>", line 1, in ? RuntimeError: Extension object missing a required method. Can you see what I'm doing wrong, or do I need to post more of the source code? -- Steven E. Harris Primus Knowledge Solutions, Inc. http://www.primus.com

2 1

just to introduce me and some comments
by Thomas.Malik＠t-online.de May 22, 2000

May 22, 2000

Hello, since i am new to this group, ok, i'm Thomas. I've taken the CXX package 3 months ago (don't know which version it was) and changed many things to suit my needs. My requirements were: - simple, homogeneous look of my wrapper classes, since i had to write a whole lot of them - use of inheritance in the wrapper classes, since the class library i had to incorporate into python, had it, and i didn't want to do mass copies of source code (every base class method in every subclass) - the whole thing has to run rock solid, since it's going to be used for interconnecting stock trading applications in a financial institute. - I wanted to rely on C++ exceptions as error notification mechanism. What i did: I moved the inheritance of the PythonExtension template from PyObject 'one level up', by introducing an intermediate class CXXPyObject with a virtual destructor: // The purpose of this is only to get a virtual destructor into a PyObject and to // be able to use dynamic_cast on it. PythonExtension inherits virtually from this. class CXXPyObject : public PyObject { protected: CXXPyObject() {} public: virtual ~CXXPyObject() {} }; CXXObject is a virtual base class for PythonExtension. Now, i can use dynamic_cast even in situations, where i inherit from PythonExtension multiple times, see below. Another thing i changed was to eliminate the template argument from PythonType. It's only purpose was to get the Object size, but you can pass this easily as a constructor argument, or even pass the size as a template argument. The downside is, you have to keep PythonExtension as a template class, since the constructor cannot determine the object size at construction time. To do easy, safe casting in the static member functions (Python entry points) i introduced a template function T python_cast<class T>(PyObject *). This does first a static_cast to the CXXObject*, and then a dynamic_cast to the object i'm actually in. Of course, this does NOT lead me magically into the paths of the right member functions of the right class, but i'm using dynamic_cast & the virtual function table for type checking, which gives me a fairly efficient type safe conversion. template<class T> T python_cast(PyObject *p) { T result = dynamic_cast<T>(static_cast< CXXPyObject* >(p)); if (result == NULL) { cerr << "!shit! cast failed!" << endl; throw bad_cast(); } return result; } Lastly, i've added some 'standard' functionality into PythonType & PythonExtension, such as str(), repr(), getattr(). I strictly obey the following rule in my source: - Python entry points always have a name starting with 's_' (such as s_str(PyObject* self, PyObject* a) ) - the first lines look like MyClassName* self=python_cast<MyClassName*>(s); // looks damn familiar, doesn't it ? Tuple args(a); so, here's some chunk out of the PythonExtension template: // since CXXPyObject is a virtual Baseclass of PythonExtension, you can // safely use multiple inheritance with the following Scheme: // class A : public PythonExtension<A> { // }; // class B : public A, public PythonExtension<B> { // } // // in A's an B's methods you may use the python_cast<A*>() thing below // to get a A* or B* out of a PyObject * template<class T> class PythonExtension: public virtual CXXPyObject { private: static void extension_object_deallocator(PyObject* t) { CXXPyObject* o=static_cast<CXXPyObject*>(t); delete o; // delete t; } explicit PythonExtension(const PythonExtension<T>& other); void operator=(const PythonExtension<T>& rhs); protected: explicit PythonExtension() { ob_refcnt = 1; ob_type = type_object(); } public: virtual ~PythonExtension() {} static PythonType& behaviors() { static PythonType* p; if(!p) { p = new PythonType(sizeof(T)); p->dealloc(extension_object_deallocator); p->str(s_str); p->repr(s_repr); } ... } If someone's interested; i can send the whole story(code). BTW, I've also implemented some huge macros for easy (and safe!) use of exceptions within C++ Modules. Note that you CAN'T throw exceptions 'through' the python interpreter. Consider this: PyObject* mainEventLoop() { PyObject* o=someObject; try { while (...) // some crufty event magic going on if (eventOccured()) Py_CallMethod(o,"crashbang",...); } catch (const Exception&) { } } and 'crashbang()' calls a C++ method from within your module, which throws an exception. My experience was that the process will go wreck, with the default execption handler aborting the program. What i did is to write a macro to catch these exceptions and to return Null instead. Ok, that's it for today. Comments are welcome. Thomas Malik

3 3

help with CXX-4.2 and add_keyword_method
by Phil Austin May 22, 2000

May 22, 2000

I'm trying to use add_keyword_method in the appended code and am getting the following errors from KCC-3.4f under Solaris 5.6. Am I guessing wrong as to the correct synax? Thanks Phil "/home/phil/include/CXX-4.2/CXX_Extensions.h", line 164: error: no instance of constructor "Py::MethodDefExt<T>::MethodDefExt [with T=testkey_module]" matches the argument list argument types are: (const char *, Py::ExtensionModule<testkey_module>::method_keyword_funct ion_t, PyObject *(PyObject *, PyObject *, PyObject *), const char *) name, ^ detected during instantiation of "void Py::ExtensionModule<T>::add_keyword_method(const char *, Py::ExtensionModule<T>::method_keyword_function_t, const char *) [with T=testkey_module]" at line 20 of "testkey.cc" #include "Python.h" #include "CXX_Objects.h" #include "CXX_Extensions.h" #include <algorithm> USING(namespace Py) USING(namespace std) extern "C" void inittestkey(); class testkey_module : public ExtensionModule<testkey_module> { public: testkey_module() : ExtensionModule<testkey_module>( "testkey" ) { add_varargs_method("sum", &testkey_module::ex_sum, "testing"); add_keyword_method("sum2", &testkey_module::ex_sum2, "testing2"); initialize( "documentation for the testkey module" ); Dict d( moduleDictionary() ); d["a_constant"] = Int(12); } virtual ~testkey_module() {} private: Object ex_sum (const Tuple &a) { cout << a[0] << endl; return a[0]; } Object ex_sum2 (const Tuple &a, const Dict &b) { cout << a[1] << endl; return a[1]; } }; void inittestkey() { static testkey_module *testkey = new testkey_module; }

3 11

Py::String should be constructible from ptr and length
by Steve Harris May 22, 2000

May 22, 2000

At present, class Py::String can be constructed from a const char* or a const std::string&, but not from const char* and a size_t. We should have a version available that takes advantage of PyString_FromStringAndSize() by taking a length argument. This would allow construction from a non-null-terminated character array (especially a subset of such an array). Also, the Py::String constructor taking a const std::string& as an argument should not use the c_str() function to call PyString_FromString(). Rather, it should call PyString_FromStringAndSize( str.data(), str.length() ) // or str.size() since we've already gone through the trouble of figuring out the length of the contained string. -- Steven E. Harris Primus Knowledge Solutions, Inc. http://www.primus.com

2 1

Re: [C++-SIG] Re: C++ extension module crashes on Solaris
by Steve Harris May 22, 2000

May 22, 2000

Harri Pasanen <harri.pasanen(a)trema.com> writes: > There is a buglet in the configure script for standard Python 1.5.2. If > gcc 2.95.2 is used on Solaris to build Python itself, -Wl,-E flags > indeed don't get passed to linker when python executable is linked. > > I've mailed a patch against this one to Guido a couple of months ago, > which he accepted. So if you grab the Python 1.5.2 from CVS the > problem should be solved. The workaround I found is to set the LDFLAGS environment variable to '-Wl,-E' before running configure. I also noticed that if you don't set the CCC environment variable to something sensible (like 'g++'), the Makefile generated by Makefile.pre.in for extension modules will fail. -- Steven E. Harris Primus Knowledge Solutions, Inc. http://www.primus.com

1 0

Re: [C++-SIG] Re: C++ extension module crashes on Solaris
by Harri Pasanen May 22, 2000

May 22, 2000

> > Message: 10 > From: Geoffrey Furnish <furnish(a)actel.com> > Date: Fri, 19 May 2000 14:35:37 -0700 (PDT) > To: Steve Harris <sharris(a)primus.com> > Cc: c++-sig(a)python.org > Subject: Re: [C++-SIG] Re: C++ extension module crashes on Solaris > <svu2fue1df.fsf(a)hodge.primus.com> > <14629.43280.621750.626706(a)zork.amer.actel.com> > <svpuqidzw7.fsf(a)hodge.primus.com> > > > Maybe you didn't catch the end of my first post. I did build my Python > > with the '-Wl,-E', but I never saw those options actually get used in > > the build. I pushed the options into my LDSHARED environment variable > > before running configure. I can tell that those options did get > > captured in the Makefile. It seems that by default the Python build > > doesn't create any shared objects, so maybe that's why my LDSHARED > > options never came into play. > > > > Should I be trying to get the '-Wl,-E' option crammed into some other > > build-time variable? I'm sorry, but I'm not sure I understand which > > parts of the system will wind up caring about those options. > > You need to make sure they got used when doing the final link step of > Python. You could check this by just rm'ing python, and then typing > make again. This will just run the final link step, and it will be > easy to see /exactly/ what got done. -Wl,-E needs to be there. If > it's not, trying just pasting in the link line by hand, adding the > options, and then seeing if this python can import your .so. > > I don't know how to influence the "standard stuff" very well, maybe > others can respond on this point. > There is a buglet in the configure script for standard Python 1.5.2. If gcc 2.95.2 is used on Solaris to build Python itself, -Wl,-E flags indeed don't get passed to linker when python executable is linked. I've mailed a patch against this one to Guido a couple of months ago, which he accepted. So if you grab the Python 1.5.2 from CVS the problem should be solved. Regards, Harri

1 0

Re: C++ extension module crashes on Solaris
by sharris＠nospam.primus.com May 22, 2000

May 22, 2000

The following message is a courtesy copy of an article that has been posted to comp.lang.python as well. Guido van Rossum <guido(a)python.org> writes: > It seems your shared library module doesn't see the symbols defined by > the Python main app. Normally, these are being exported by default. > I wonder if the trick of using g++ for the linker breaks this? Yes, that turned out the to be the problem. I will take some time on Monday to prepare a more complete summary of what I learned (based upon _lots_ of helpful contributions), but to summarize: * Python must be linked with the option '-Wl,-E' passed to the linker. (That is, on Solaris, using g++, with the GNU binutils package.) Other advice involving compiling the extension with '-fPIC' also seemed relevant, but turned out to not make any difference. Now I'm not sure why '-fPIC' _isn't_ required! -- Steven E. Harris Primus Knowledge Solutions, Inc. http://www.primus.com

1 0

Re: [C++-SIG] Re: C++ extension module crashes on Solaris
by Steve Harris May 20, 2000

May 20, 2000

"Lyle Johnson" <jlj(a)cfdrc.com> writes: [...] > I use these flags to compile the individual files: > > g++ -fPIC -c file1.cpp > g++ -fPIC -c file2.cpp > > and then for the link step: > > g++ -Wl,-E -shared -o seh_test.so file1.o file2.o ... > > I noticed in particular that you are not using the "-fPIC" flag for > the compile step. I know that this is REQUIRED on most platforms for > this to work properly, so it might be the missing piece of the > puzzle in your case. I didn't recompile Python yet, but I did rebuild my extension module and got a similar, but different result: $ make g++ -fPIC -I../Include -g -O2 \ -I/users/sharris/usr/local/include/python1.5 -DHAVE_CONFIG_H \ -c ./pod.cc g++ -fPIC -I../Include -g -O2 \ -I/users/sharris/usr/local/include/python1.5 -DHAVE_CONFIG_H \ -c ./pypod.cc g++ -fPIC -I../Include -g -O2 \ -I/users/sharris/usr/local/include/python1.5 -DHAVE_CONFIG_H \ -c ./seh_test.cc g++ -fPIC -I../Include -g -O2 \ -I/users/sharris/usr/local/include/python1.5 -DHAVE_CONFIG_H \ -c ./../Src/cxx_extensions.cxx gcc -I../Include -g -O2 \ -I/users/sharris/usr/local/include/python1.5 -DHAVE_CONFIG_H \ -c ./../Src/cxxextensions.c g++ -fPIC -I../Include -g -O2 \ -I/users/sharris/usr/local/include/python1.5 -DHAVE_CONFIG_H \ -c ./../Src/cxxsupport.cxx g++ -shared -Wl,-E \ pod.o pypod.o seh_test.o \ cxx_extensions.o cxxextensions.o cxxsupport.o \ -o seh_test.so $ cp seh_test.so ~/usr/local/lib/python1.5/ $ python Python 1.5.2 (#1, May 19 2000, 09:34:45) [GCC 2.95.2 19991024 (release)] on sunos5 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import seh_test Traceback (innermost last): File "<stdin>", line 1, in ? ImportError: ld.so.1: python: fatal: relocation error: file ./seh_test.so: symbol PyExc_RuntimeError: referenced symbol not found >>> Note that this time, it's the symbol PyExc_RuntimeError that's causing the problem. Does it matter that the gcc line above (for ../Src/cxxextensions.c) doesn't have the 'fPIC' option on it? -- Steven E. Harris Primus Knowledge Solutions, Inc. http://www.primus.com

3 2

Re: [C++-SIG] Re: C++ extension module crashes on Solaris
by Steve Harris May 19, 2000

May 19, 2000

Geoffrey Furnish <furnish(a)actel.com> writes: [...] > This case is different. you aren't building a shared lib, and your > link line doesn't link your extension module against any shared libs. > Instead, you are expecting to pick up your missing external symbols by > having them bound against the symbols in the enclsoing process > context. i.e., Python itself. I find the situation here a little weird too, but I'm trying to follow the "supported Python way" by using the Setup file and Makefile.pre.in, which apparently advocates this system. > In order for this to work, some systems (I don't know whether Slowaris > is like this or not, but some are, so it might be) require that the > host application must "export symbols" so that the dynamic loader can > consider them candidates for symbol resolution when dso's are > dynloaded. > > In other words: > link your Python with "-Wl,-E" (but it doesn't have to > be compiled -fPIC) > Compile your extension module with -fPIC, lijnke with -shared. > > It seems you're now doing this last step, but I don't think we've seen > from the info presented earlier if you're doing the first step. Maybe you didn't catch the end of my first post. I did build my Python with the '-Wl,-E', but I never saw those options actually get used in the build. I pushed the options into my LDSHARED environment variable before running configure. I can tell that those options did get captured in the Makefile. It seems that by default the Python build doesn't create any shared objects, so maybe that's why my LDSHARED options never came into play. Should I be trying to get the '-Wl,-E' option crammed into some other build-time variable? I'm sorry, but I'm not sure I understand which parts of the system will wind up caring about those options. -- Steven E. Harris Primus Knowledge Solutions, Inc. http://www.primus.com

2 1

Re: [C++-SIG] Re: C++ extension module crashes on Solaris
by Steve Harris May 19, 2000

May 19, 2000

Phil Austin <phil(a)geog.ubc.ca> writes: [...] > It also might be worth checking to see whether your copy of python > really does have PyExc_IndexError defined: $ nm ~/usr/local/bin/python | egrep 'IndexError' [2971] | 704664| 4|OBJT |GLOB |0 |19 |PyExc_IndexError Yep, got that. I'm going to go try the fPIC option. Must Python be recompiled with that option, or just my extension module? -- Steven E. Harris Primus Knowledge Solutions, Inc. http://www.primus.com

1 0