Mailman 3 February 2003 - Cplusplus-sig

Redesign for to_python/from_python converters
by David Abrahams May 21, 2003

May 21, 2003

Some background for this thread is in the following messages: http://article.gmane.org/gmane.comp.python.c++/638 describes the global Boost.Python to/from-python converter registry which associates conversion code with the C++ type(s) being converted. On re-reading this message, I realize it's confusingly worded. Don't be discouraged if it doesn't all sink in. The point is that when a function accepting a C++ argument of type Foo is wrapped, it needs a procedure to extract a Foo from the Python object. The registry associates the code for that procedure with the type Foo. http://article.gmane.org/gmane.comp.python.c++/2044 describes a proposed scheme for dealing with the fact that currently, any given C++ type can have only one to-python conversion in the global registry. This is a problem because one module may use class_<vector<string> >, while another one registers a custom conversion from vector<string> to a Python tuple of strings. http://article.gmane.org/gmane.comp.python.c%2B%2B/2161 describes the general procedure for defining an rvalue from_python converter, and shows why I haven't yet documented how to make user-defined conversions: it's just too hairy a procedure in v2 at the moment. The purpose of this thread is to discuss the design of a new interface for users to interact with to/from-python converters. First, I'd like to discuss some things which I think are desirable. It would be worth knowing which of these are important to the community: 1. Users should have the option to say that certain to/from-python converters shall only have a local effect in a single extension module. 2. It should be possible to optimize away the cost of registry lookups in some cases where it is known that the conversion is defined in the local extension module. This is a completely separate issue from #1. We can have either one without the other. 3. It should be easy for users to explicitly define new conversions. Let's deal with #3 first. You might ask, "what was wrong with the old Boost.Python v1 approach? It sure was simple!" PyObject* to_python(SomeUDT const& x) { ... } SomeUDT from_python(PyObject* p, boost::type<SomeUDT>) { ... } It was simple, but it was basically not legal C++. It worked in so many places because it exploited a very common (and somewhat subtle) bug in C++ compilers, but when if you tried to use more conforming compilers (e.g. CodeWarrior >= 8 or recent EDGs), it would fail to compile. The reasons have to do with the rules for looking up these functions from within templates. Such a scheme can be made to work legally, but you have to resort to weird tricks like asking users to define all their converters before #including Boost.Python headers or adding dummy arguments to the functions for the sake of argument-dependent lookup. There is another problem with the v1 scheme. In particular, it was an important design goal of Boost.Python v2 to eliminate the use of C++ exceptions as part of the process of resolving overloaded C++ functions (in v1, we would throw a special exception to indicate a Python argument couldn't be converted to the corresponding C++ argument type). That means that from_python conversion has to be a 2-phase process: first, determine whether a conversion is possible, then if all arguments can be converted, do all the conversions and call the C++ function. Any user-defined from_python conversion needs to be able to report convertibility separately from actually doing the conversion. There were also some ways in which the v1 interface was hard to use: the 2nd argument to the from_python converter had to exactly match the argument type to any C++ functions being wrapped, so you might need: SomeUDT from_python(PyObject* p, boost::type<SomeUDT>) { ... } SomeUDT& from_python(PyObject* p, boost::type<SomeUDT&>) { ... } SomeUDT const& from_python(PyObject* p, boost::type<SomeUDT const&>) { ... } SomeUDT* from_python(PyObject* p, boost::type<SomeUDT*>) { ... } SomeUDT const* from_python(PyObject* p, boost::type<SomeUDT const*>) { ... } and a few others. So what should the interface look like? Let's first examine the constraints that the C++ language imposes. User-defined converters can be viewed as behavioral customizations of templates in the Boost.Python library. We basically have two approaches available to us for customizing behaviors: 1. Runtime dispatching through virtual functions or function pointers. This is the approach currently taken in Boost.Python v2 for most converters. The converter registry contains pointer to functions which implement the conversions. Some runtime dispatching is always needed for cross-module conversion support, unless you want to repeat the conversion code in every module which needs it (and nobody wants that). There are other reasons to do this having to do with the way dynamic_cast and RTTI are implemented in most compilers. 2. Compile-time customization. This was the default approach in Boost.Python v1: the compiler would look up the appropriate to_/from_python function and insert a call in the function wrapper. It could even be completely inlined. The disadvantage of this approach is that it is heaviliy dependent on code visibility: the customizations have to be visible in every place that wants to take advantage of them. Boost.Python v2 uses compile-time customization only for to-python conversions of "builtin" C++ types for which there can be only one reasonable Python interpretation. For example, const char* and std::string both are converted to Python strings. Ideally, the user could select compile-time customization for to_python conversion of selected types, and additionally choose to export their converters (e.g. to the global registry). I'm really unsure about the value of compile-time customization of from_python converters. The biggest problem with it is that it limits the whole extension module to *one* conversion method for a given C++ type. Normally, a from_python converter is used to convert one Python type to a given C++ type. If another extension module has exported conversions for other Python types to the same C++ type, do you want to be able use them? There are basically two viable techniques for compile-time customization in C++: defining functions to be found by argument-dependent lookup, and template specialization. For various reasons I won't bore you with, I feel that template specialization is the only recourse for Boost.Python. I'll begin discussion of some possible interfaces in a follow-on message. -- David Abrahams dave(a)boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution

2 15

passing Numeric arrays as arguments
by Greg Landrum May 19, 2003

May 19, 2003

[Boost 1.29, python 2.2.2, g++ 3.2, RH8.0] I've been trying to get a Numeric array object passed in as an argument to a function I've defined in a BPLv2 module, without much success. At the moment I'm happy to get the object in as a PyArrayObject *, because I've got a bunch of code that is expecting those pointers. Since I can't figure out how to extract a PyArrayObject * from a python::numeric::array (if that's even possible), I've attempted to define the following simple function: PyObject *MatFunc(PyArrayObject *arg){ PyObject *res = PyTuple_New(2); return res; } (eventually this will do more, I'm starting small) but whenever I call this Python with a Numeric array as an argument, I get the dreaded catch-all "TypeError: bad argument type for built-in operation". Am I wrong to be even attempting this? Should I just give up and do a normal extension module? -greg ---- greg Landrum (greglandrum(a)earthlink.net) Software Carpenter/Computational Chemist

11 27

Re: [C++-sig] VC7 + auto_ptr + transfer of owership
by Nicolas Lelong March 17, 2003

March 17, 2003

>auto_ptr support is only really complete in the current CVS >version. There are some problems with it in 1.29.0. Huhf, I'm afraid that I checkouted the yesterday's version, and that's the one that causes the problem ! Any idea ? Thanks, Nicolas _________________________________________________________________ Protect your PC - get McAfee.com VirusScan Online http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963

3 6

Boost.Python compiles & links on Darwin
by Nicholas K. Sauter March 6, 2003

March 6, 2003

This should be of interest to people waiting to use Boost.Python on the Macintosh. I have achieved partial success running Boost.Python modules on Darwin. The build system is based on the scheme proposed by Rene Rivera last November, with the Boost.Python library dynamically linked as a dylib, and the user's module as a bundle. Python itself must be installed as a Darwin Framework. It was also necessary to download gcc 3.2.2 from FSF rather than use the pre-installed compiler from Apple. Details are posted in the "darwin_notes" link presented on my Multi-platform build page: http://cci.lbl.gov/boost As noted, this is only a partial port. For one thing, the gcc 3.2.2 compiler doesn't seem to properly handle any C++ exceptions, so many of the modules in the Boost.Python test suite will fail. It may be necessary to wait for a proper gcc 3.3 release to fix this. Also, some work will be needed on the Boost.Python library itself to fix some compile-time problems reported in the log files (see build page). -- Nicholas K. Sauter, Ph.D. Lawrence Berkeley National Lab 1 Cyclotron Road, Bldg 4R0230 Berkeley, CA 94720-8235 nksauter(a)lbl.gov Voice: 510-486-5713 Fax: 510-486-5909

3 4

Inherited instance variables
by Matt Giger March 5, 2003

March 5, 2003

Hi, I just discovered Boost.Python and I am very impressed (as I am with most of Boost). I have embedded a Python interpreter in my project and would like to tie in some C++ classes but am having problems with inherited member variables. Here is my example: C++ --- struct foo { foo(void) {} virtual ~foo(void) {} virtual bool handle_evt(object event) { return false; } list children; } struct foowrap : foo { foowrap(PyObject* self) : self_(self) {} foowrap(PyObject* self, const foo& f) : foo(f), self_(self) {} bool handle_evt(object event) { return call_method<bool>(self_, "handle_evt", event); } static bool def_handle_evt(foo* f, object event) { return f->foo::handle_evt(event); } PyObject* const self_; }; BOOST_PYTHON_MODULE(foomod) { class<foo, foowrap>("foo") .def("handle_evt", &foowrap::def_handle_evt) .def_readwrite("children", &widget::children); } Python ------ import foomod class subfoo(foomod.foo): def __init__(self, parent): self.parent = parent self.parent.children.append(self) def __del__(self) self.parent.children.remove(self) def handle_evt(self, event) print event for child in children: if handle_evt(event) return true else return false f = foo() # works fine g = subfoo(f) # works fine h = subfoo(g) # breaks Error: self.parent.children.append(self) TypeError: bad argument type for built-in operation # no children list available in the subfoo class instance 'g' print g.children TypeError: bad argument type for built-in operation It appears that the children list object is not visible from any subclasses of the foo class. I have tried changing the line: .def_readwrite("children", &widget::children) with .add_property("children", make_getter(&foo::children, return_internal_reference<>()), make_setter(&foo::children)) but still no luck...any help would be appreciated, thanks. -- Matt Giger Lunar Software, Inc. mgiger(a)lunarsoft.com

2 2

Enhanced shared_ptr support
by David Abrahams March 2, 2003

March 2, 2003

A few days ago I checked in some changes to the way shared_ptrs get converted to Python. A while back I had added a special shared_ptr deleter which caused all shared_ptr conversions from python to maintain the lifetime of the owning Python object, not of the C++ object. In other words, we'll have this: shared_ptr<T> T +------+------+ +------+ | | | | | | * | *------->| | | || | | | | +--||--+------+ +------+ || /\ \/ || +-----------------+ || | | || | Python Object *======++ | | +-----------------+ The double lines represent ownership relationships. Now, when such a shared_ptr is converted back to python, the owned Python object is returned, rather than a new Python object being built around the shared_ptr, sharing ownership of the T object with the Python object above. -Dave -- Dave Abrahams Boost Consulting www.boost-consulting.com

2 1

BPL embedding tutorial updated.
by Dirk Gerrits Feb. 28, 2003

Feb. 28, 2003

I've made some changes to my Boost.Python tutorial on embedding. Joel and I will hopefully be able to integrate it into the main tutorial for the next Boost release. In the meantime it is available from the Boost sandbox: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/boost-sandbox/boo… If you find any errors or have any suggestions, I'd love to hear from you. Dirk Gerrits

1 0

extracting a c++ object from a PyObject
by Francois Ostiguy Feb. 27, 2003

Feb. 27, 2003

This is a follow-up question to a recent thread. Dave Abrahams provided the following code to extract a c++ object from a PyObject void f(PyObject* p) { handle<> ph(borrowed(p)); MyObj* op = extract<MyObj*>(object(ph)); } this works beautifully, but appenrently fails in the case where MyObj is a simple function. That is typedef void (fcntptr*)(); void f(PyObject* p) { handle<> ph(borrowed(p)); fcntptr op = extract<fcntptr>(object(ph)); } results in the following compilation error message. /usr/local/include/boost/type_traits/add_cv.hpp:34: `const volatile' qualifiers cannot be applied to `void ()()' As always, any help/insight is greatly appreciated. -Francois ---------------------------------------------------------------------------- Dr. Jean-Francois OSTIGUY voice: (630) 840-2231 Beam Physics Dept MS220 FAX: (630) 840-6039 Fermi National Accelerator Laboratory email: ostiguy(a)fnal.gov Batavia IL 60510-0500 WWW:www-ap.fnal.gov/~ostiguy

2 1

problem with python::dict[]
by harold fellermann Feb. 26, 2003

Feb. 26, 2003

Hello, I got your address from http://mail.python.org/pipermail/c++-sig/2002-June/001311.html -- so I don't know whether you are the right person to ask. I think I have found an error in the boost.python's dict-interface. My following code-snippet runs into segmantation fault: #include <stdlib.h> #include <boost/python/dict.hpp> using namespace boost::python; int main() { dict x[750]; return 0; } The problem seems to concern the garbage collector from the Python/C-API, as my debugger tells me, the program crashes in _PyObject_GC_New(): (gdb) run Starting program: /home/tsb/harold/c_network/pytest [New Thread 1024 (LWP 25905)] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1024 (LWP 25905)] 0x0809a909 in PyErr_Occurred () (gdb) bt #0 0x0809a909 in PyErr_Occurred () #1 0x080addca in _PyObject_GC_New () #2 0x08064edc in PyDict_New () #3 0x4003b1f4 in boost::python::detail::dict_base::dict_base() () from /usr/local/lib/libboost_python.so.1.29.0 #4 0x080c6c6c in main () at pytest.c:9 #5 0x401744a2 in __libc_start_main () from /lib/libc.so.6 I tried this with a couple of python-objects and all containers (dict, list, ...) crashed in exactly the same way. Interestingly the program behaves normal when I reduce the number of objects within my array to approx. 700. My configuration looks as follows: - python 2.2.1 (#1, Sep 10 2002, 17:49:17) - gcc 3.2 - boost version 1.29.0 Do you have any idea what is going on here? Is this a known bug, maybe even a nowadays fixed one? Or is it only a question of configuration? Any help will be appreciated! thanx, - harold fellermann - -- He who wonders discovers that this in itself is wonder. -- M.C. Escher

2 1

Virtual Methods + default arguments
by Bruno da Silva de Oliveira Feb. 26, 2003

Feb. 26, 2003

Hi! Is it possible to indicate default arguments for virtual methods, using the BOOST_PYTHON_MEMBER_FUNCTION_OVERLOADS macro? I have this code: struct X { virtual void f(int x = 10, char c = 'a') { std::cout << x << ", " << c << std::endl; } }; void call_f(X& x) { x.f(); } BOOST_PYTHON_MEMBER_FUNCTION_OVERLOADS(f_overloads, f, 0, 2) struct XWrap: X { XWrap(PyObject* self_): self(self_) {} XWrap(PyObject* self_, const X&): self(self_) {} PyObject* self; void f(int x = 10, char c = 'a') { call_method<void>(self, "f", x, c); } void default_f(int x = 10, char c = 'a') { X::f(x, c); } }; BOOST_PYTHON_MODULE(test) { def("call_f", &call_f); class_< X, XWrap >("X") .def("f", &X::f, &XWrap::default_f, f_overloads()) ; } But this does not compile. Here's the first line of the output of the compiler: D:\Programming\Libraries\boost-cvs\boost\boost/python/detail/caller.hpp(53): error: class "f_overloads" has no member "result_converter" So I tried to change the class_ definition to this: class_< X, XWrap >("X") .def("f", &X::f, &XWrap::default_f) .def("f", &X::f, f_overloads()) ; It compiles fine, but in python: >>> from test import * >>> x = X() >>> x.f(10, 'c') Traceback (most recent call last): File "<stdin>", line 1, in ? RuntimeError: unidentifiable C++ exception >>> I also tried to change the second .def to: .def("f", &XWrap::f, f_overloads()) and: .def("f", &XWrap::default_f, f_overloads()) But with the same results. So, is it possible to have virtual methods with default arguments? If so, how? Any pointer would be greatly appreciated. Thanks in advance, Nicodemus.

3 2