Mailman 3 December 2009 - Cplusplus-sig

boost::python embedding error - runtime - Mac OS X - 1.38
by Igor Karpov Oct. 1, 2014

Oct. 1, 2014

Hi, I am trying to run the following program: #include <iostream> #include <Python.h> #include <boost/python.hpp> using namespace boost::python; using namespace std; int main(int argc, char** argv) { { // Using Python/C Py_Initialize(); PyObject* main_module = PyImport_ImportModule("__main__"); PyObject* globals = PyEval_GetGlobals(); PyObject* locals = PyEval_GetLocals(); PyRun_SimpleString("print 'Hello World, from Python/C!'\n"); Py_Finalize(); } { // Using boost::python Py_Initialize(); object main_module = import("__main__"); object main_namespace = main_module.attr("__dict__"); try { object ignored = exec("print 'Hello World, from boost::python!'\n", main_namespace); Py_Finalize(); } catch (error_already_set const& e) { PyErr_Print(); return 1; } } return 0; } on Linux (both x86 and x86_64), with Ubuntu's default version of Boost (1.34.1) installed, I am able to do so as expected: $ ./embed Hello World, from Python/C! Hello World, from boost::python! on Mac OS X, I get a runtime error: $ ./embed Hello World, from Python/C! Bus Error Running with gdb, this is the stack trace: Hello World, from Python/C! Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) where #0 0x00000000 in ?? () #1 0x00bb618f in PyEval_GetGlobals () #2 0x00bce0dd in PyImport_Import () #3 0x00bce2b0 in PyImport_ImportModule () #4 0x008127d4 in boost::python::import (name=@0xbffff4a8) at libs/python/src/import.cpp:20 #5 0x00004446 in main (argc=1, argv=0xbffff510) at embed.cc:22 For this Mac, boost version is latest (1.38) and python version is: Python 2.5.4 (r254:67917, Dec 23 2008, 14:57:27) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin Is this a bug? I attached the source and the cmake file I am using to build, for convenience. Thanks, --Igor.

4 6

CMake and getting starting with boost.python
by Braddock Gaskill Aug. 6, 2011

Aug. 6, 2011

Yesterday I became a boost.python user, after years of heavy SWIG usage. I would have started using boost.python years ago except for the bjam learning curve. I spent hours TRYING on multiple occasions, but got lost every time trying to get the page one example to build with bjam. Here is how to build the "greet" example with a four line CMake file: cat >>CMakeLists.txt <<EOF LINK_LIBRARIES(boost_python) INCLUDE_DIRECTORIES("/usr/include/python2.5") LINK_DIRECTORIES("/usr/lib/python2.5") # Not needed? ADD_LIBRARY(hello SHARED hello.cpp) EOF You WILL have to `mv libhello.so hello.so` after building. Works for me under Ubuntu Linux 6.10 after apt-get install cmake python2.5-dev boost-python-dev cmake make mv libhello.so hello.so ipython In [1]: import hello In [2]: hello.greet() Out[2]: 'hello, world' Boost.Python has been a joy to learn and use since I got past bjam. I had a moderately large API wrapped in just my first couple hours of learning (less time than I spent messing with bjam). Any additional advice for using CMake to build boost.python? Braddock Gaskill

4 4

How to import a boost::python dll in windows?
by ichijoji Aug. 5, 2011

Aug. 5, 2011

I'm trying to port some research code that uses boost::python to expose a library to python to windows, but I'm having trouble importing the dll into python. I'm compiling using msvc 2005 and using python 2.5, but when I try to import the module I get this: Traceback (most recent call last): File "main.py", line 1, in <module> import object File "c:\code\sge\code\scripts\object.py", line 1, in <module> import sge ImportError: No module named sge My module is named sge and I'm compiling to sge.dll. Is there some obvious reason why python wouldn't be able to find my dll? When I compile on linux into an so this code works fine. -- View this message in context: http://www.nabble.com/How-to-import-a-boost%3A%3Apython-dll-in-windows--tf3… Sent from the Python - c++-sig mailing list archive at Nabble.com.

4 4

Getting object for PyObject (reference counting)
by Murray Cumming Feb. 11, 2010

Feb. 11, 2010

I can't find any definitive documentation that tells me how I should get a boost::python::object to wrap an existing PyObject*. I guess that there's a way to do it that uses an existing reference, and a way that takes an extra reference. -- murrayc(a)murrayc.com www.murrayc.com www.openismus.com

5 10

Implementation of proper overload resolution
by Troy D. Straszheim Jan. 6, 2010

Jan. 6, 2010

Here's what I've got on overloading. This turned out to be a lot more work, and this mail much longer, than I'd hoped. Would appreciate a readthrough and comments/questions, I've done about all I can do. First a review of the problems, then a walkthrough of an implementation I have that fixes them. 1.a) Problem: Ambiguous calls from python to overloaded functions Boost.python's 'overloading' is and has always been broken w.r.t. is ambiguity. The library currently has no notion, given some tuple of arguments, of one function being a better match than another. It knows only "this function matches" or not. What the user observes is the following: when a call to a c++ function is made from python e.g. >>> m.f(1) boost.python looks through the overloads for f, in the reverse order of registration, and calls the first one that is callable. For instance if the registrations are void f_int(int); void f_bool(bool); BOOST_PYTHON_MODULE(m) { def("f", f_int); def("f", f_bool); } then f_bool will execute this call of f(1), since the python 'int' converts to c++ bool. If the overloads were registered in the reverse order, f_int would be executed. In this case the c++ overloads in question are potentially distinguishable from one another, as python has distinct types bool and int. So these aren't "necessarily" ambiguous. 1.b) Problem: "Necessarily" ambiguous overloads There are some overload sets that *are* necessarily ambiguous. This set, for instance, void f_double(double); void f_float(float); BOOST_PYTHON_MODULE(m) { def("f", f_double); def("f", f_float); } can never be unambiguous, since python has no 'float' type. I.e. >>> f(x) will call f_float for any value of x where the call succeeds at all. 1.c) Problem: Multiple values for keyword argument At the same place in the boost.python code where this 'first-match' overload resolution is done, the user error 'multiple values for keyword argument' is not checked. Neal Becker recently pointed this out. With boost.python it looks like this: int addem(int x, int y, int z) { return x*100 + y*10 + z; } BOOST_PYTHON_MODULE(M) { def("addem", &addem, (arg("x"), arg("y"), arg("z"))); } >>> from M import addem >>> addem(1, 8, 2, x=4) Traceback (most recent call last): ... ArgumentError: Python argument types in M.addem(int, int, int) did not match C++ signature: addem(int x, int y, int z) That error message is very confusing... f(int,int,int) doesn't match f(int,int,int)? The pure python version reports something more sensible: >>> def g(x,y,z, **kwargs): ... print x,y,z,kwargs ... >>> g(1,2,3,x=4) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: g() got multiple values for keyword argument 'x' 2.) An implemented solution I've got something implemented. Here's what it does. 2.a) Solution: multiple values for keyword The easiest case to catch is the last [1.c]. It is also orthogonal to the others. If a registered function has keyword arguments, check for multiple keyword values, and raise an error if so: >>> from M import addem >>> addem(1,2,3,x=1) Boost.Python.TypeError: M.addem() got multiple values for keyword argument 'x' 2.b) Solution: "necessarily" ambiguous overloads The next easiest thing to catch is case [1.b], "necessarily" ambiguous registrations. It proceeds as follows: at import time, as each new overload V for function F is registered, compare V to the existing overloads EO for F. If V has the same signature as something in EO, raise an AmbiguousOverload exception. For instance, if you load the module from [1.b] above, you get: >>> import m Traceback (most recent call last): ... AmbiguousOverload: Boost.Python function m.f has ambiguous overloads. C++ signatures f(float) -> None and f(double) -> None are indistinguishable to python. Again this is because c++ float and c++ double both map to python 'float'. This one is "easy" as it happens only once (not at every function call) and doesn't take up any space. 2.c) Solution: ambiguous calls The hard case is [1.a]: void f_bool(bool); void f_int(int); BOOST_PYTHON_MODULE(m) { def(f, f_bool); def(f, f_int); } For module 'm' above, a call to f(True) or f(1) should succeed and call the corresponding function. Passing a float, however, is ambiguous: >>> f(True) # ok >>> f(1) # ok >>> f(1.0) Traceback (most recent call last): File "<stdin>", line 1, in <module> Boost.Python.AmbiguousCall: Ambiguous call to Boost.Python function m.f C++ signatures: f(int) f(bool) So the implementation has some how 'scored' each possible overload and in each case either used the best one or reported ambiguity if multiple overloads are tied for 'best'. The scoring works as follows. In the guts of boost.python we recieve a tuple of arguments of type PyObject*. We also have a list of overloads, each with an mpl vector representing the signature of the associated C++ function. This is unchanged from the released version. What has been done until now is to attempt to call each overload in order and return the result of the first that succeeds. In this new implmentation, for each c++ argument of type T_n, we ask a class overload_score<T_n> to score the conversion of the type (PyTypeObject*) of this PyObject* to T_n. Example: template <> struct overload_score<int> { boost::optional<unsigned> operator()(PyObject* type) { if(PyInt_CheckExact(type)) return boost::optional<unsigned>(0); // int == perfect else if (PyBool_Check(type)) return boost::optional<unsigned>(1); // bool == okay else if (PyFloat_CheckExact(type)) return boost::optional<unsigned>(1); // float == okay else if(arg_from_python<int>(type).convertible()) return boost::optional<unsigned>(1); // fallback else return boost::optional<unsigned>(); // unsuitable } }; The "score" type is optional<unsigned>. A valueless, (default-constructed) optional<unsigned> means 'unsuitable'. optional<unsigned>(0) is a perfect match, and optional<unsigned>(value) for value > 0 is a workable but less than perfect match. These per-argument scores are added together for all arguments: this is the overload's total score. If any argument is 'unsuitable', the total score is 'unsuitable'. If there is a tie for the best (lowest) score, the call is ambiguous and a Boost.Python.AmbiguousCall exception is raised. If there is only one function in first place, call it. 3.) Implications This breaks a few corner cases that I've found in the tests. 3.a) implied init<> registration I have found a few instances like this one: struct X { X(); X(int); }; class_<X>("X") // #1 .def(init<>()) // #2 .def(init<int>()) ; Here, #1 causes a default constructor to be registered, as does #2. This will cause a throw at load time as in [2.b]. It is simple to fix. 3.b) raw_constructor The test suite for raw_constructor depends on our first_match overload resolution. The fix is to move a few things around to get the same effect without relying on this behavior. The "before" code looks like this: class Foo { public: Foo(tuple args, dict kw) : args(args), kw(kw) {} tuple args; dict kw; }; object init_foo(tuple args, dict kw) { tuple rest(args.slice(1,_)); return args[0].attr("__init__")(rest, kw); } BOOST_PYTHON_MODULE(raw_ctor_ext) { // using no_init postpones defining __init__ function until after // raw_function for proper overload resolution order, since later // defs get higher priority. class_<Foo>("Foo", no_init) .def("__init__", raw_function(&init_foo)) .def(init<tuple, dict>()) .def_readwrite("args", &Foo::args) .def_readwrite("kw", &Foo::kw) ; } The "after" code does the registration piecemeal: Maybe there is a better way to do this. To my mind this is a little better because it makes explicit what is happening rather than relying on some subtle property of overload resolution: class Foo { public: Foo(tuple args, dict kw) : args(args), kw(kw) {} tuple args; dict kw; }; // our custom factory function object init_foo(tuple args, dict kw) { tuple rest(args.slice(1,_)); return args[0].attr("__real_init__")(rest, kw); } BOOST_PYTHON_MODULE(raw_ctor_ext) { // to get the desired effect we register // the actual constructor and our 'raw' constructor, // and then rename them class_<Foo> c("Foo", init<tuple, dict>()); c .def("__tmp_init__", raw_function(&init_foo)) .def_readwrite("args", &Foo::args) .def_readwrite("kw", &Foo::kw) ; // // __init__ => __real_init__ // __tmp_init__ => __init__ // object real_constructor = getattr(c, "__init__"); object raw_constructor = getattr(c, "__tmp_init__"); setattr(c, "__init__", raw_constructor); delattr(c, "__tmp_init__"); setattr(c, "__real_init__", real_constructor); } And that basically covers it. Looking forward to comments/feedback. The code is here: http://gitorious.org/~straszheim/boost/straszheims-python/trees/master Note that the headers are in subdirectory 'include'... If there is enough response to this mail it I'll send out instructions on how to get a working build going. - t

6 11

Re: [C++-sig] [python] python + phoenix
by lists_ravi＠lavabit.com Jan. 6, 2010

Jan. 6, 2010

>> I have some code that takes member function pointers of the form >> R Obj::*( A1, A2, A3, ..., An ) >> and converts them to a function object Q with the following signature: >> Rnew Q( Obj&, B1, B2, ..., Bn ) >> where >> Rnew = result_conversion_metafunction<R>::type >> Bi = arg_conversion_metafunction<Ai>::type >> and Q is exposed via boost.python as a member of Obj. >> >> For example, Ai could be a fixed-point number, with Bi being a double so >> that >> the python side does not know anything about fixed-point numbers. > > I gather that custom converters aren't preferred because: > > 1. It is a hassle to talk directly to the underlying raw PyObject > pointers and manage storage in the converter methods > > 2. The converted type still leaks out to python in docstrings > > Are there others? Or am I completely off? Neither of the above is a big issue. In fact, python-side knowledge of the converted type would even be mildly useful. The main problem is the number of types: my_fixed_point_type< int bit_width, // 1 to 63 int binary_pt_location, // -63 to 63 bool signed, property_tag // policies for rounding, etc. > std::complex< my_fixed_point_type<...> > boost::numeric::ublas::array< ... > boost::numeric::ublas::matrix< ... > There are simply far too many types to create converters for and to register explicitly. Therefore, I use inline functions/functors to do all the conversions on the C++ side, and python knows only about real and complex doubles. > > An instance >> of Q would convert eh floating point numbers passed from the python side >> into >> fixed-point numbers, call the member function, and convert the returned >> value >> to a double for use on the python side. >> >> For these "converters", metafunctions from boost.function_types are used >> to >> obtain the mpl::vector specifying result type and argument types. >> Extending >> the technique above to function objects which take Obj* as their first >> argument, I have a protocol which relies on the presence of a typedef >> called >> 'components' in the function object so that I can use the converter when >> exposing via boost.python: >> >> struct my_functor { >> typedef mpl::vector<R, Obj*, A1, A2, A3, A4> components; >> R operator()( Obj*, A1, A2, A3, A4 ) { ... } >> }; > > To be sure I'm following you, correct me: > > * for each member function signature that you wrap, you have a matching > my_functor that calls it > * my_functor does nothing but call the member function it wraps > * my_converter<my_functor>::type does the conversion from Rnew (B1, B2, > ... Bn) to R (A1, A2, ...An) and calls my_functor. > > Have I got it? Yes. > It is all very interesting, and to a certain degree duplicates some > functionality in detail/caller.hpp. How do you generate these > my_functors... preprocessor? fusion? Mostly boost.preprocessor (with some template metaprogramming using mpl). Fusion doesn't work very well for me since the effort needed to get my functors into a form usable from fusion is greater than just coding my own invocation functions via boost.preprocessor; pseudo-code: template <typename Func> struct my_converter { Func f_; typedef typename ft::components<F>::type components; typedef typename arg_converter<components>::type arg_types; typedef typename result_converter<components>::type result_type; // Shown for non-void return with one argument; can be generalized // using enable_if for void types and using boost.preprocessor for // different numbers of arguments result_type operator()( typename at_c<1,arg_types>::type a1 ) { return convert_result<result_type>( f_( convert_arg<typename at_c<1,components>::type>( a1 ) ) ); } }; template <typename Func> my_converter<Func> make_converted_func( Func f ) { return my_converter<Func>( f ); } > I'm wondering if there isn't motivation here to cleanly integrate a > general facility for additional c++-side type conversions. The > following came to mind, which imitates a function type style that > boost::proto uses extensively: > > struct hidden_type; // python shouldn't see this at all > > struct int_2_hidden // converts python-side 'int' to hidden > { > typedef hidden_type result_type; > > hidden_type operator()(int) const; > }; > > // fnobj takes a hidden, doesn't know it is wrapped in python > struct fnobj { > typedef void result_type; > void operator()(float, hidden_type); > }; > > def("myfunctor", as<void(float, int_to_hidden(int))>(fnobj())); > > where int_to_hidden(int) is a function type (not a function call, but it > later becomes a function call), indicating that what python passes > should be converted to int, then the int converted to hidden_type via an > instance of int_to_hidden, then the hidden_type passed to the underlying > instance of fnobj. > > I realize this doesn't involve using the mpl::vectors you've already > calculated, just throwing it out there. Such a general facility would probably be better than my code and would be greatly useful for me, but are there enough people out there with this use case? I don't care about using mpl::vector if there is a more general solution. >> // my_converter uses the components typedef >> typedef typename my_converter<my_functor>::type Q; > > so Q could have a nested typedef (note I say Rnew, not R): > > typedef mpl::vector<Rnew, Obj*, B1, B2, B3, B4> components; > > or use that in combination with function_types to synthesize > > typedef Rnew(Obj*, B1, B2, B3, B4) signature; Yes. Regards, Ravi

3 3

[Py++] Export #define directives to ctypes
by Nikolaus Rath Dec. 28, 2009

Dec. 28, 2009

Hello, Is there a way to export preprocessor definitions (#define ...) with the ctypes code generator? At the moment I'm using this rather nasty hack: /* Header file */ // FUSE_SET_ATTR_* is defined with #define in fuse_lowlevel.h #include <fuse_lowlevel.h> const int D_FUSE_SET_ATTR_MODE = FUSE_SET_ATTR_MODE; const int D_FUSE_SET_ATTR_UID = FUSE_SET_ATTR_UID; // ... # Code generator mb.write_module(os.path.join(code_path, 'ctypes_api.py')) fh = open(os.path.join(code_path, 'ctypes_api.py'), 'a') fh.write('\n') for var in mb.global_ns.decls(lambda f: f.name.startswith('D_')): fh.write('{0} = {1}\n'.format(var.name[2:], var._value)) fh.close() But it'd be nice if there was a better solution (so that I don't have to define a new variable for every #define that I want to export). Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C

2 4

[Py++] ctypes generator ignores bit fields
by Nikolaus Rath Dec. 27, 2009

Dec. 27, 2009

Hello, The ctypes code generator translates the struct struct fuse_file_info { int flags; unsigned long fh_old; int writepage; unsigned int direct_io : 1; unsigned int keep_cache : 1; unsigned int flush : 1; unsigned int padding : 29; uint64_t fh; uint64_t lock_owner; }; into fuse_file_info._fields_ = [ #class fuse_file_info ("flags", ctypes.c_int), ("fh_old", ctypes.c_ulong), ("writepage", ctypes.c_int), ("direct_io", ctypes.c_uint), ("keep_cache", ctypes.c_uint), ("flush", ctypes.c_uint), ("padding", ctypes.c_uint), ("fh", ctypes.c_ulonglong), ("lock_owner", ctypes.c_ulonglong), ] but this is wrong, the correct translation is: fuse_file_info._fields_ = [ ('flags', ctypes.c_int), ('fh_old', ctypes.c_ulong), ('writepage', ctypes.c_int), ('direct_io', ctypes.c_uint, 1), ('keep_cache', ctypes.c_uint, 1), ('flush', ctypes.c_uint, 1), ('padding', ctypes.c_uint, 29), ('fh', ctypes.c_uint64), ('lock_owner', ctypes.c_uint64)] Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C

2 6

[Py++] Don't transform char* to c_char_p
by Nikolaus Rath Dec. 27, 2009

Dec. 27, 2009

Hi Roman, I know you are busy, so I have already monkeypatched pyplusplus locally for my purposes. But maybe you are interested to add this feature add some point in the future: It would be great if there was a way to tell Py++ that it should not represent the C char * type to the ctypes c_char_p, because the later is supposed to be used only for \0 terminated strings. A general char* should be translated to POINTER(c_char). Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C

2 4

[Py++] ctypes generator produces wrong Python code for function pointers in structs
by Nikolaus Rath Dec. 26, 2009

Dec. 26, 2009

Hello, Py++ translates the struct typedef int cb_fun_t(char *); struct info { cb_fun_t* cb_fun; int flag; }; into the Python Structure class info(ctypes.Structure): """class info""" info._fields_ = [ #class info ("cb_fun", ctypes.POINTER( ctypes.CFUNCTYPE( ctypes.c_int, ctypes.c_char_p ) )), ("flag", ctypes.c_int), ] This is wrong and produces segfaults. The correct version is class info(ctypes.Structure): """class info""" info._fields_ = [ #class info ("cb_fun", ctypes.CFUNCTYPE( ctypes.c_int, ctypes.c_char_p ) ), ("flag", ctypes.c_int), ] I have attached a small example that demonstrates the problem. lib.c is a small C library, build_api.py creates the broken api.py with Py++. api_correct.py is the manually corrected API with the correct structure. call.py demonstrates that the first API segfaults and the second API works: $ python call.py Calling do_callback with flag=7 do_callback called with flag=7 cb_fun received 'Hello World' cb_fun returns 42 cb_fun returned 42 do_callback returned 42 Calling do_callback with flag=7 do_callback called with flag=7 Segmentation fault Please let me know if you can reproduce the problem. It took me quite some time to figure this out. Best, -Nikolaus -- »Time flies like an arrow, fruit flies like a Banana.« PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C

2 1