Mailman 3 March 2003 - Cplusplus-sig

Redesign for to_python/from_python converters
by David Abrahams May 21, 2003

May 21, 2003

Some background for this thread is in the following messages: http://article.gmane.org/gmane.comp.python.c++/638 describes the global Boost.Python to/from-python converter registry which associates conversion code with the C++ type(s) being converted. On re-reading this message, I realize it's confusingly worded. Don't be discouraged if it doesn't all sink in. The point is that when a function accepting a C++ argument of type Foo is wrapped, it needs a procedure to extract a Foo from the Python object. The registry associates the code for that procedure with the type Foo. http://article.gmane.org/gmane.comp.python.c++/2044 describes a proposed scheme for dealing with the fact that currently, any given C++ type can have only one to-python conversion in the global registry. This is a problem because one module may use class_<vector<string> >, while another one registers a custom conversion from vector<string> to a Python tuple of strings. http://article.gmane.org/gmane.comp.python.c%2B%2B/2161 describes the general procedure for defining an rvalue from_python converter, and shows why I haven't yet documented how to make user-defined conversions: it's just too hairy a procedure in v2 at the moment. The purpose of this thread is to discuss the design of a new interface for users to interact with to/from-python converters. First, I'd like to discuss some things which I think are desirable. It would be worth knowing which of these are important to the community: 1. Users should have the option to say that certain to/from-python converters shall only have a local effect in a single extension module. 2. It should be possible to optimize away the cost of registry lookups in some cases where it is known that the conversion is defined in the local extension module. This is a completely separate issue from #1. We can have either one without the other. 3. It should be easy for users to explicitly define new conversions. Let's deal with #3 first. You might ask, "what was wrong with the old Boost.Python v1 approach? It sure was simple!" PyObject* to_python(SomeUDT const& x) { ... } SomeUDT from_python(PyObject* p, boost::type<SomeUDT>) { ... } It was simple, but it was basically not legal C++. It worked in so many places because it exploited a very common (and somewhat subtle) bug in C++ compilers, but when if you tried to use more conforming compilers (e.g. CodeWarrior >= 8 or recent EDGs), it would fail to compile. The reasons have to do with the rules for looking up these functions from within templates. Such a scheme can be made to work legally, but you have to resort to weird tricks like asking users to define all their converters before #including Boost.Python headers or adding dummy arguments to the functions for the sake of argument-dependent lookup. There is another problem with the v1 scheme. In particular, it was an important design goal of Boost.Python v2 to eliminate the use of C++ exceptions as part of the process of resolving overloaded C++ functions (in v1, we would throw a special exception to indicate a Python argument couldn't be converted to the corresponding C++ argument type). That means that from_python conversion has to be a 2-phase process: first, determine whether a conversion is possible, then if all arguments can be converted, do all the conversions and call the C++ function. Any user-defined from_python conversion needs to be able to report convertibility separately from actually doing the conversion. There were also some ways in which the v1 interface was hard to use: the 2nd argument to the from_python converter had to exactly match the argument type to any C++ functions being wrapped, so you might need: SomeUDT from_python(PyObject* p, boost::type<SomeUDT>) { ... } SomeUDT& from_python(PyObject* p, boost::type<SomeUDT&>) { ... } SomeUDT const& from_python(PyObject* p, boost::type<SomeUDT const&>) { ... } SomeUDT* from_python(PyObject* p, boost::type<SomeUDT*>) { ... } SomeUDT const* from_python(PyObject* p, boost::type<SomeUDT const*>) { ... } and a few others. So what should the interface look like? Let's first examine the constraints that the C++ language imposes. User-defined converters can be viewed as behavioral customizations of templates in the Boost.Python library. We basically have two approaches available to us for customizing behaviors: 1. Runtime dispatching through virtual functions or function pointers. This is the approach currently taken in Boost.Python v2 for most converters. The converter registry contains pointer to functions which implement the conversions. Some runtime dispatching is always needed for cross-module conversion support, unless you want to repeat the conversion code in every module which needs it (and nobody wants that). There are other reasons to do this having to do with the way dynamic_cast and RTTI are implemented in most compilers. 2. Compile-time customization. This was the default approach in Boost.Python v1: the compiler would look up the appropriate to_/from_python function and insert a call in the function wrapper. It could even be completely inlined. The disadvantage of this approach is that it is heaviliy dependent on code visibility: the customizations have to be visible in every place that wants to take advantage of them. Boost.Python v2 uses compile-time customization only for to-python conversions of "builtin" C++ types for which there can be only one reasonable Python interpretation. For example, const char* and std::string both are converted to Python strings. Ideally, the user could select compile-time customization for to_python conversion of selected types, and additionally choose to export their converters (e.g. to the global registry). I'm really unsure about the value of compile-time customization of from_python converters. The biggest problem with it is that it limits the whole extension module to *one* conversion method for a given C++ type. Normally, a from_python converter is used to convert one Python type to a given C++ type. If another extension module has exported conversions for other Python types to the same C++ type, do you want to be able use them? There are basically two viable techniques for compile-time customization in C++: defining functions to be found by argument-dependent lookup, and template specialization. For various reasons I won't bore you with, I feel that template specialization is the only recourse for Boost.Python. I'll begin discussion of some possible interfaces in a follow-on message. -- David Abrahams dave(a)boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution

2 15

passing Numeric arrays as arguments
by Greg Landrum May 19, 2003

May 19, 2003

[Boost 1.29, python 2.2.2, g++ 3.2, RH8.0] I've been trying to get a Numeric array object passed in as an argument to a function I've defined in a BPLv2 module, without much success. At the moment I'm happy to get the object in as a PyArrayObject *, because I've got a bunch of code that is expecting those pointers. Since I can't figure out how to extract a PyArrayObject * from a python::numeric::array (if that's even possible), I've attempted to define the following simple function: PyObject *MatFunc(PyArrayObject *arg){ PyObject *res = PyTuple_New(2); return res; } (eventually this will do more, I'm starting small) but whenever I call this Python with a Numeric array as an argument, I get the dreaded catch-all "TypeError: bad argument type for built-in operation". Am I wrong to be even attempting this? Should I just give up and do a normal extension module? -greg ---- greg Landrum (greglandrum(a)earthlink.net) Software Carpenter/Computational Chemist

11 27

pyste and big files
by Niki Spahiev April 8, 2003

April 8, 2003

I got msvc internal overflow with pyste generated files. Can pyste write separate finctions for each class? Niki Spahiev

2 2

pyste and virtual classes.
by Giulio Eulisse April 4, 2003

April 4, 2003

It's my understanding that if I exclude() some virtual function out of a pystified class, pyste generates a code which is not compilable because the compiler complains about abstract functions being present in the wrapper class as wrapper methods for excluded one are not generate by pyste. I think it would be nice to have some wrapper code being generated anyway(maybe just throwing some exception) so that one can exclude safely virtual methods out of a class interface. Ciao, Giulio

3 12

Pyste typedef problems...
by Giulio Eulisse April 1, 2003

April 1, 2003

Tring to pystify ---CUT-HERE--- typedef unsigned int B; class A { B b; public: B &f(void){return b;}; }; ---CUT-HERE--- with pyste file: ---CUT-HERE--- A=Class("A","A.h") ---CUT-HERE--- gets to the following error, when compiling the output c++ code... /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/object/make_instance.hpp: In static member function `static PyObject* boost::python::objects::make_instance_impl<T, Holder, Derived>::execute(Arg&) [with Arg = unsigned int*, T = unsigned int, Holder = boost::python::objects::pointer_holder<unsigned int*, unsigned int>, Derived = boost::python::objects::make_ptr_instance<unsigned int, boost::python::objects::pointer_holder<unsigned int*, unsigned int> >]': /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/to_python_indirect.hpp:62: instantiated from `static PyObject* boost::python::detail::make_reference_holder::execute(T*) [with T = unsigned int]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/detail/unwind_type.hpp:19: instantiated from `Generator::result_type boost::python::detail::unwind_type_cv(U*, boost::python::detail::cv_tag<false, false>, Generator*) [with Generator = boost::python::detail::make_reference_holder, U = unsigned int]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/detail/unwind_type.hpp:48: instantiated from `Generator::result_type boost::python::detail::unwind_ptr_type(U*, Generator*) [with Generator = boost::python::detail::make_reference_holder, U = unsigned int]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/detail/unwind_type.hpp:58: instantiated from `static Generator::result_type boost::python::detail::unwind_helper<is_ptr>::execute(U, Generator*) [with Generator = boost::python::detail::make_reference_holder, U = unsigned int*, bool is_ptr = true]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/detail/unwind_type.hpp:77: instantiated from `Generator::result_type boost::python::detail::unwind_type(const U&, Generator*) [with Generator = boost::python::detail::make_reference_holder, U = unsigned int*]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/detail/unwind_type.hpp:26: instantiated from `Generator::result_type boost::python::detail::unwind_type_cv(const U*, boost::python::detail::cv_tag<true, false>, Generator*) [with Generator = boost::python::detail::make_reference_holder, U = unsigned int]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/detail/unwind_type.hpp:48: instantiated from `Generator::result_type boost::python::detail::unwind_ptr_type(U*, Generator*) [with Generator = boost::python::detail::make_reference_holder, U = const unsigned int]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/detail/unwind_type.hpp:69: instantiated from `static Generator::result_type boost::python::detail::unwind_helper<false>::execute(U&, Generator*) [with Generator = boost::python::detail::make_reference_holder, U = const unsigned int]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/detail/unwind_type.hpp:77: instantiated from `Generator::result_type boost::python::detail::unwind_type(const U&, Generator*) [with Generator = boost::python::detail::make_reference_holder, U = unsigned int]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/to_python_indirect.hpp:103: instantiated from `PyObject* boost::python::to_python_indirect<T, MakeHolder>::operator()(T) const [with T = B&, MakeHolder = boost::python::detail::make_reference_holder]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/detail/invoke.hpp:93: instantiated from `PyObject* boost::python::detail::invoke(boost::python::detail::mem_fn_tag, RC*, F&, TC&) [with RC = boost::python::to_python_indirect<B&, boost::python::detail::make_reference_holder>, F = B&(A::*)(), TC = boost::python::detail::nullary<boost::python::arg_from_python<A&> >]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/detail/caller.hpp:162: instantiated from `PyObject* boost::python::detail::caller_arity<1>::impl<F, ConverterGenerators, Policies, Sig>::operator()(PyObject*, PyObject*) [with F = B&(A::*)(), ConverterGenerators = boost::python::detail::args_from_python, Policies = boost::python::return_internal_reference<1, boost::python::default_call_policies>, Sig = boost::mpl::list2<B&, A&>]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/function/function_template.hpp:117: instantiated from `static R boost::detail::function::function_obj_invoker2<FunctionObj, R, T0, T1>::invoke(boost::detail::function::any_pointer, T0, T1) [with FunctionObj = boost::python::detail::caller<B&(A::*)(), boost::python::detail::args_from_python, boost::python::return_internal_reference<1, boost::python::default_call_policies>, boost::mpl::list2<B&, A&> >, R = PyObject*, T0 = PyObject*, T1 = PyObject*]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/function/function_template.hpp:481: instantiated from `void boost::function2<R, T0, T1, Allocator>::assign_to(FunctionObj, boost::detail::function::function_obj_tag) [with FunctionObj = boost::python::detail::caller<B&(A::*)(), boost::python::detail::args_from_python, boost::python::return_internal_reference<1, boost::python::default_call_policies>, boost::mpl::list2<B&, A&> >, R = PyObject*, T0 = PyObject*, T1 = PyObject*, Allocator = std::allocator<boost::function_base>]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/function/function_template.hpp:432: instantiated from `void boost::function2<R, T0, T1, Allocator>::assign_to(Functor) [with Functor = boost::python::detail::caller<B&(A::*)(), boost::python::detail::args_from_python, boost::python::return_internal_reference<1, boost::python::default_call_policies>, boost::mpl::list2<B&, A&> >, R = PyObject*, T0 = PyObject*, T1 = PyObject*, Allocator = std::allocator<boost::function_base>]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/function/function_template.hpp:293: instantiated from `boost::function2<R, T0, T1, Allocator>::function2(Functor, boost::detail::function::enable_if<boost::type_traits::ice_not<boost::is_same<Functor, int>::value>::value, int>::type) [with Functor = boost::python::detail::caller<B&(A::*)(), boost::python::detail::args_from_python, boost::python::return_internal_reference<1, boost::python::default_call_policies>, boost::mpl::list2<B&, A&> >, R = PyObject*, T0 = PyObject*, T1 = PyObject*, Allocator = std::allocator<boost::function_base>]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/make_function.hpp:61: instantiated from `boost::python::api::object boost::python::detail::make_function_aux(F, const CallPolicies&, const ConverterGenerators&, const Sig&, const boost::python::detail::keyword_range&, NumKeywords) [with F = B&(A::*)(), CallPolicies = boost::python::return_internal_reference<1, boost::python::default_call_policies>, ConverterGenerators = boost::python::detail::args_from_python, Sig = boost::mpl::list2<B&, A&>, NumKeywords = boost::mpl::int_<0>]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/make_function.hpp:90: instantiated from `boost::python::api::object boost::python::make_function(F, const CallPolicies&, const Keywords&) [with F = B&(A::*)(), CallPolicies = boost::python::return_internal_reference<1, boost::python::default_call_policies>, Keywords = boost::python::detail::keywords<0>]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/class.hpp:395: instantiated from `void boost::python::class_<T, X1, X2, X3>::def_impl(const char*, Fn, const Helper&, ...) [with Fn = B&(A::*)(), Helper = boost::python::detail::def_helper<boost::python::return_internal_reference<1, boost::python::default_call_policies>, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>, T = A, X1 = boost::python::detail::not_specified, X2 = boost::python::detail::not_specified, X3 = boost::python::detail::not_specified]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/class.hpp:459: instantiated from `void boost::python::class_<T, X1, X2, X3>::def_maybe_overloads(const char*, Fn, const A1&, ...) [with Fn = B&(A::*)(), A1 = boost::python::return_internal_reference<1, boost::python::default_call_policies>, T = A, X1 = boost::python::detail::not_specified, X2 = boost::python::detail::not_specified, X3 = boost::python::detail::not_specified]' /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/class.hpp:253: instantiated from `boost::python::class_<T, X1, X2, X3>& boost::python::class_<T, X1, X2, X3>::def(const char*, A1, const A2&) [with A1 = B&(A::*)(), A2 = boost::python::return_internal_reference<1, boost::python::default_call_policies>, T = A, X1 = boost::python::detail::not_specified, X2 = boost::python::detail::not_specified, X3 = boost::python::detail::not_specified]' typedef_confusion.cpp:14: instantiated from here /afs/cern.ch/sw/lcg/external/Boost/1.30.0/rh73_gcc32/boost/python/object/make_instance.hpp:23: `sizeof' applied to incomplete type `boost::STATIC_ASSERTION_FAILURE<false>' Any idea? Ciao, Giulio

3 5

unnamed enum
by Giulio Eulisse April 1, 2003

April 1, 2003

Hi, when pystifing qcolor.h from QT 3.0.5, there is an unnamed enum: class QColor{ [...] enum { Dirt = 0x44495254, Invalid = 0x49000000 }; [...] }; This gets translated by pyste as enum_< QColor::._0 >("._0") .value("Invalid", QColor::Invalid) .value("Dirt", QColor::Dirt) ; Which is obviously not valid because of ._0...Is this a known bug? Any idea on how to mask/exclude it? Ciao, Giulio -- Giulio Eulisse Northeastern University, Boston, MA (USA) CERN, Geneve, GE (CH) Office: 40-3-A15 Office phone:76 71692 Cellphone:(+39) 3356972726

3 6

operator !=?
by Giulio Eulisse March 31, 2003

March 31, 2003

When trying to pystify any of the operator!= of any of the QT classes I get the following error: boostQT.cc: In function `void init_module_boostQT ()': boostQT.cc:1458: no matching function for call to `boost::python::class_<QVariant, boost::python::detail::not_specified, boost::python::detail::not_specified, boost::python::detail::not_specified>::def (bool)' boostQT.cc:1914: parse error before `.' boostQT.cc:2914: no matching function for call to `boost::python::class_<QWidget, boost::noncopyable, {unnamed}::QWidget_Wrapper, boost::python::detail::not_specified>::def (const char[15], {unknown type}, void ({unnamed}::QWidget_Wrapper::*) (int, int))' /usr/include/g++-3/stl_relops.h: In function `bool operator!= (const _Tp &, const _Tp &) [with _Tp = boost::python::self_ns::self_t]': boostQT.cc:1458: instantiated from here /usr/include/g++-3/stl_relops.h:38: no match for `!boost::python::detail::operator_<op_eq, boost::python::self_ns::self_t, boost::python::self_ns::self_t>' /usr/include/g++-3/stl_relops.h:38: candidates are: operator!(bool) <builtin> It happens, for example, with QVariant: class QVariant { [...] bool operator==( const QVariant& ) const; [...] }; which gets converted to: .def( self != self ) which looks correct to me...Any idea? -- Giulio Eulisse Northeastern University, Boston, MA (USA) CERN, Geneve, GE (CH) Office: 40-3-A15 Office phone:76 71692 Cellphone:(+39) 3356972726 Homepage: http://home.cern.ch/eulisse/

3 2

Arrays and Boost.Python
by Patrick Hartling March 30, 2003

March 30, 2003

Does Boost.Python (from Boost 1.30.0 or the Boost CVS trunk) provide handling of arrays as public data members in a C++ class? Using Boost.Python from Boost 1.30.0, I get an error from GCC 3.2 about line 68 of boost/python/data_members.hpp stating that "ISO C++ forbids assignment of arrays." If I change the array definition in my Boost.Python code so that the data member is read-only instead of read/write, that gets rid of the error. I don't mind doing that in this case--really I am just curious about how Boost.Python handles C++ arrays. -Patrick -- Patrick L. Hartling | Research Assistant, VRAC patrick(a)vrac.iastate.edu | 2624 Howe Hall: 1.515.294.4916 http://www.137.org/patrick/ | http://www.vrac.iastate.edu/

2 3

How to exclude a function from being exported using pyste.
by Giulio Eulisse March 30, 2003

March 30, 2003

Hi, I've a class like this: class A { public: A(int x); A(float x); int b(int x); int b(float x); }; And I want to export to python only A::b(int x), not A::b(float x). How do I do it using pyste? A = Class("A","A.h") exclude(A.b) excludes BOTH. Same problem with constructors: how do I get rid of only one of the two constructors? Ciao, Giulio

4 4

bug with const double and msvc
by Niki Spahiev March 29, 2003

March 29, 2003

Commented lines in attached code procude errors with msvc toolset. There is workaround which i can use thanks to pyste. I patched it to always cast member pointers and then removed all consts from const double. Niki Spahiev P.S. pyste is great even without gcc-xml. class test1 { public: void f1( const double a, const double b, const double c ) {} void f2( const int a, const double b ) {} }; class test2 { public: void f1( const double a, const double b, const double c ) {} void f1( const int a, const double b ) {} }; // Includes ==================================================================== #include <boost/python.hpp> // Using ======================================================================= using namespace boost::python; // Module ====================================================================== BOOST_PYTHON_MODULE(tt) { class_< test1, boost::noncopyable > c1("test1"); //~ c1.def("f1", &test1::f1); /* ERROR */ c1.def("f1", (void (test1::*)( double, double, double )) &test1::f1); class_< test2, boost::noncopyable > c2("test2"); //~ c2.def("f1", (void (test2::*)( const double, const double, const double )) &test2::f1); /* ERROR */ c2.def("f1", (void (test2::*)( double, double, double )) &test2::f1); } // Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 12.00.8804 for 80x86 ...updating 4 targets... vc-C++ ..\..\..\libs\python\pyste\bin\tt.pyd\msvc\release\runtime-link-dynamic\bbug.obj bbug.cpp C:\code\BOOST_~2\boost/python/make_function.hpp(94) : error C2665: 'get_signature' : none of the 4 overloads can convert parameter 1 from type 'void (__thiscall test1::*)(const double,const double,const double)' C:\code\BOOST_~2\boost/python/class.hpp(400) : see reference to function template instantiation 'class boost::python::api::object __cdecl boost::python::make_function(void (__thiscall test1::*)(const double,const double,const double),const struct boost::python::default_call_policies &,const struct boost::python::detail::keywords<0> &)' being compiled C:\code\BOOST_~2\boost/python/make_function.hpp(97) : error C2780: 'class boost::python::api::object __cdecl boost::python::detail::make_function_aux(F,const CallPolicies &,const ConverterGenerators &,const Sig &)' : expects 4 arguments - 6 provided C:\code\BOOST_~2\boost/python/make_function.hpp(29) : see declaration of 'make_function_aux' C:\code\BOOST_~2\boost/python/class.hpp(400) : see reference to function template instantiation 'class boost::python::api::object __cdecl boost::python::make_function(void (__thiscall test1::*)(const double,const double,const double),const struct boost::python::default_call_policies &,const struct boost::python::detail::keywords<0> &)' being compiled C:\code\BOOST_~2\boost/python/make_function.hpp(94) : error C2665: 'get_signature' : none of the 4 overloads can convert parameter 1 from type 'void (__thiscall test2::*)(const double,const double,const double)' C:\code\BOOST_~2\boost/python/class.hpp(400) : see reference to function template instantiation 'class boost::python::api::object __cdecl boost::python::make_function(void (__thiscall test2::*)(const double,const double,const double),const struct boost::python::default_call_policies &,const struct boost::python::detail::keywords<0> &)' being compiled C:\code\BOOST_~2\boost/python/make_function.hpp(97) : error C2780: 'class boost::python::api::object __cdecl boost::python::detail::make_function_aux(F,const CallPolicies &,const ConverterGenerators &,const Sig &)' : expects 4 arguments - 6 provided C:\code\BOOST_~2\boost/python/make_function.hpp(29) : see declaration of 'make_function_aux' C:\code\BOOST_~2\boost/python/class.hpp(400) : see reference to function template instantiation 'class boost::python::api::object __cdecl boost::python::make_function(void (__thiscall test2::*)(const double,const double,const double),const struct boost::python::default_call_policies &,const struct boost::python::detail::keywords<0> &)' being compiled

2 2