I'm trying to update the PyOSG wrappers to allow access to an array type returned from a C++ class method. All attempts to call a Python method on the returned array or it's elements (even print it), result in a Boost.Python.ArgumentError (error msg at bottom). Here's the wrap of the array class (osg::Vec3Array): namespace PyOSG { void init_ArrayVec3() { TemplateCompArray<osg::Vec3Array, osg::Vec3> Vec3Array("Vec3Array"); } } and here's a piece of the referenced template: using namespace boost::python; template <typename T, typename BT> class TemplateCompArray { public : TemplateCompArray(const char * name) : _array(name, boost::python::no_init) { _array .def(init<>()) .def(init<int>()) .def("__init__", make_constructor(&create_from_tuple)) .def("getNumElements", &T::getNumElements) .def("__getitem__", &getitem, return_internal_reference<>()) .def("__setitem__", &setitem, return_internal_reference<>()) .def("resize", &resize) .def("__str__", &str) ... static std::string str(T * self) { ... } } Any idea why printing (or calling a Python method of the wrapped Vec3Array in Python should generate a type error? It's like boost.python doesn't realize the TemplateCompArray instantiation "is" Vec3Array. Thanks, Randall P.S. If you want to see the full source for context, check out: http://mosca.caltech.edu/pyosg_downloads/pyosg_ads_15.tar.gz and look in osg/ArrayVec3.cpp and include/TemplateCompArray.hpp. ------------------------------------------------------------------------------ Traceback (most recent call last): ... File "osg_to_sgg.py", line 147, in setupGeode print geom.getVertexArray() Boost.Python.ArgumentError: Python argument types in Vec3Array.__str__(Vec3Array) did not match C++ signature: __str__(osg::TemplateArray<osg::Vec3f, (osg::Array::Type)10, 3, 5126>*)
Left out a useful piece: Randall Hopper: |and here's a piece of the referenced template: | | using namespace boost::python; | | template <typename T, typename BT> | class TemplateCompArray | { | public : | TemplateCompArray(const char * name) : _array(name, boost::python::no_init) { | _array | .def(init<>()) | .def(init<int>()) | .def("__init__", make_constructor(&create_from_tuple)) | .def("getNumElements", &T::getNumElements) | .def("__getitem__", &getitem, return_internal_reference<>()) | .def("__setitem__", &setitem, return_internal_reference<>()) | .def("resize", &resize) | .def("__str__", &str) | ... | static std::string str(T * self) | { | ... | } private: class_<T, osg::ref_ptr<T>, bases<osg::Array>, boost::noncopyable> _array; | } I guess I've been perplexed a bit by the boost syntax and not knowing how the boost templates expand and what that does to the type structure. But I'm starting to suspect that some sort of type redirection is required since the actual boost-wrapped array is a private member of the registered class. I'm I even close? Thanks, Randall
I've been digging into this and can summarize the problem more succinctly now (I was barking up the wrong tree before). What do I need to do to tell boost::python to dereference a wrapped C++ pointer before trying to invoke the C++ class instance's methods? I've got a method returning a pointer to a C++ class instance, and right now I can't use it as a callable because boost::python treats it like a pointer. Traceback (most recent call last): ... File "osg_to_sgg.py", line 153, in setupGeode v[0].set( 0, 1, 2 ) Boost.Python.ArgumentError: Python argument types in Vec3Array.__getitem__(Vec3Array, int) did not match C++ signature: __getitem__(osg::TemplateArray<osg::Vec3f, (osg::Array::Type)10, 3, 5126>*, int) Note that the Vec3Array type is identical to the osg::TemplateArray<...> type (via typedef), so the key difference is the extra "*" in the last line. Thanks, Randall
On 11/4/06, Randall Hopper <viznut@charter.net> wrote:
I've been digging into this and can summarize the problem more succinctly now (I was barking up the wrong tree before).
What do I need to do to tell boost::python to dereference a wrapped C++ pointer before trying to invoke the C++ class instance's methods?
I've got a method returning a pointer to a C++ class instance, and right now I can't use it as a callable because boost::python treats it like a pointer.
Traceback (most recent call last): ... File "osg_to_sgg.py", line 153, in setupGeode v[0].set( 0, 1, 2 ) Boost.Python.ArgumentError: Python argument types in Vec3Array.__getitem__(Vec3Array, int) did not match C++ signature: __getitem__(osg::TemplateArray<osg::Vec3f, (osg::Array::Type)10, 3, 5126>*, int)
Note that the Vec3Array type is identical to the osg::TemplateArray<...> type (via typedef), so the key difference is the extra "*" in the last line.
I don't think anyone is able to follow you. In my opinion, you'd better create one message with clear description of the problem + small test case that reproduces the problem. Take into account, that most of the people here do not know what is osg. -- Roman Yakovenko C++ Python language binding http://www.language-binding.net/
Roman Yakovenko: |Randall Hopper; |> What do I need to do to tell boost::python to dereference a wrapped |> C++ pointer before trying to invoke the C++ class instance's methods? | |I don't think anyone is able to follow you. Thanks for the candid reply. I thought I had a simple question, but I'll see what I can do about thining this down to a simple example. The problem is I'm trying to fix someone else's wrappers, and after hours reading the boost docs and playing with this problem, I don't understand the fundamentals of boost enough to "get under the hood" (dump the compiler output, and trace metaprogramming templates). Despite the selling point, boost "does" require it's own metalanguage, and it's as confusing as hell to the newbie that needs to leave the yellow brick road of: .def( "f", &f, "Hi mom" ) I really do want to become skilled at boost, so I'd greatly appreciate it if you'd point me at 5-10 URLs to read in order (in case I haven't already read them). As far as experience level, I'm an accomplished C++ programmer (15+ years), but I don't usually use templates to solve problems because in C++ they often obfuscate more than they simplify. I'm willing to delve in though. |In my opinion, you'd better create one message with clear description of |the problem + small test case that reproduces the problem. Take into |account, that most of the people here do not know what is osg. Sorry, it's not that relevent but "osg" is the namespace prefix used by OpenSceneGraph (www.openscenegraph.org). The boost::python wrappers for OpenSceneGraph I'm working on are here: http://mosca.caltech.edu/pyosg_downloads/pyosg_ads_15.tar.gz Thanks, Randall
Randall Hopper <viznut@charter.net> writes:
Roman Yakovenko: |Randall Hopper; |> What do I need to do to tell boost::python to dereference a wrapped |> C++ pointer before trying to invoke the C++ class instance's methods? | |I don't think anyone is able to follow you.
Thanks for the candid reply. I thought I had a simple question, but I'll see what I can do about thining this down to a simple example.
This should be pretty simple.
The problem is I'm trying to fix someone else's wrappers, and after hours reading the boost docs and playing with this problem, I don't understand the fundamentals of boost enough to "get under the hood" (dump the compiler output, and trace metaprogramming templates).
Why in anyone's name would you want to do that?
Despite the selling point, boost "does" require it's own metalanguage, and it's as confusing as hell to the newbie that needs to leave the yellow brick road of:
.def( "f", &f, "Hi mom" )
I really do want to become skilled at boost, so I'd greatly appreciate it if you'd point me at 5-10 URLs to read in order (in case I haven't already read them). As far as experience level, I'm an accomplished C++ programmer (15+ years), but I don't usually use templates to solve problems because in C++ they often obfuscate more than they simplify.
I think you're scapegoating the technology; the information needed to debug your problem was in your previous message; it has nothing to do with metaprogramming or templates. -- Dave Abrahams Boost Consulting www.boost-consulting.com
Randall Hopper <viznut@charter.net> writes:
I've been digging into this and can summarize the problem more succinctly now (I was barking up the wrong tree before).
What do I need to do to tell boost::python to dereference a wrapped C++ pointer
wrapped how?
before trying to invoke the C++ class instance's methods?
Normally -- if you've exposed a C++ class X to Python with class_<X, ...>("PyX") -- when you ask Boost.Python to convert an X* to Python, it complains because it doesn't know whether and how to manage the lifetime of the referenced X object. Normally one uses a CallPolicy (http://tinyurl.com/yjt72m#python.call_policies) to describe how the pointer should be handled. Specifically, it's usually an return_value_policy<R> where R is a ResultConverterGenerator (http://tinyurl.com/ydmvuf#result_converter_generators) such as manage_new_object. Once you've succeeded in making the conversion, what you end up with on the Python side is just a PyX object that happens to get to the X instance through that pointer, and operations on the converted X* work just the same as if you had converted an X object directly to Python.
I've got a method returning a pointer to a C++ class instance, and right now I can't use it as a callable because boost::python treats it like a pointer.
I don't think that's the problem.
Traceback (most recent call last): ... File "osg_to_sgg.py", line 153, in setupGeode v[0].set( 0, 1, 2 ) Boost.Python.ArgumentError: Python argument types in Vec3Array.__getitem__(Vec3Array, int)
__getitem__ refers to the square brackets operator, so this has nothing to do with callability of the Vec3Array object.
did not match C++ signature: __getitem__(osg::TemplateArray<osg::Vec3f, (osg::Array::Type)10, 3, 5126>*, int) Note that the Vec3Array type is identical to the osg::TemplateArray<...> type (via typedef), so the key difference is the extra "*" in the last line.
Interesting; how did you generate that __getitem__ method? Its first parameter should be a reference to a Vec3Array, not a pointer. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams <dave@boost-consulting.com> writes:
Traceback (most recent call last): ... File "osg_to_sgg.py", line 153, in setupGeode v[0].set( 0, 1, 2 ) Boost.Python.ArgumentError: Python argument types in Vec3Array.__getitem__(Vec3Array, int)
__getitem__ refers to the square brackets operator, so this has nothing to do with callability of the Vec3Array object.
did not match C++ signature: __getitem__(osg::TemplateArray<osg::Vec3f, (osg::Array::Type)10, 3, 5126>*, int) Note that the Vec3Array type is identical to the osg::TemplateArray<...> type (via typedef), so the key difference is the extra "*" in the last line.
Interesting; how did you generate that __getitem__ method? Its first parameter should be a reference to a Vec3Array, not a pointer.
Come to think of it, the reason we prefer a reference is that wrapped C++ functions with pointer parameters will accept a wrapped NULL pointer, and those accepting a reference parameter will automatically reject a NULL pointer at the library level, saving the function author from the NULL check. So the signature of the function wrapped to generate that __getitem__ method above is *more* permissive than if it was built in the usual correct manner. If you're certain that osg::TemplateArray<osg::Vec3f, (osg::Array::Type)10, 3, 5126> and Vec3Array are identical types, then I'm stumped. I'd like to see the code that exposes Vec3Array to Python. But even more than that, I'd like to see a minimal, reproducible example of your problem. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams: |> What do I need to do to tell boost::python to dereference a wrapped |> C++ pointer before trying to invoke the C++ class instance's methods? | |Specifically, it's usually an return_value_policy<R> where R is a |ResultConverterGenerator such as manage_new_object. | |Once you've succeeded in making the conversion, what you end up with |on the Python side is just a PyX object that happens to get to the X |instance through that pointer, and operations on the converted X* work |just the same as if you had converted an X object directly to Python. Thanks for the reply, Dave. This code does use a return policy ("manage_osg_object", specifically), as this C++ package uses its own ref counting: http://mosca.caltech.edu/pyosg_downloads/pyosg_ads_15.tar.gz Off and on today, I've been paring this package down to the absolute minimum needed to reproduce the problem so it hopefully will take less than a minute of someone's time to point out what I'm missing. I hope to post that soon. I'm determined to get to the bottom of this. For what boost::python does do when merely given function pointer, it is "truly" amazing. I'd really like to add it to my tool belt. I just haven't found the secret decoder ring yet. ;-) |> I've got a method returning a pointer to a C++ class instance, and |> right now I can't use it as a callable because boost::python treats it like |> a pointer. | |I don't think that's the problem. Ok. |> Traceback (most recent call last): |> ... |> File "osg_to_sgg.py", line 153, in setupGeode |> v[0].set( 0, 1, 2 ) |> Boost.Python.ArgumentError: Python argument types in |> Vec3Array.__getitem__(Vec3Array, int) | |__getitem__ refers to the square brackets operator, so this has |nothing to do with callability of the Vec3Array object. Thanks, I realized that. The boost-wrapped Vec3Array defines it's own __getitem__ method (see below). |> did not match C++ signature: |> __getitem__(osg::TemplateArray<osg::Vec3f, (osg::Array::Type)10, 3, 5126>*, int) |> Note that the Vec3Array type is identical to the osg::TemplateArray<...> |> type (via typedef), so the key difference is the extra "*" in the last |> line. | |Interesting; how did you generate that __getitem__ method? Its first |parameter should be a reference to a Vec3Array, not a pointer. I'll post a complete working example soon, but here are the relevent snippets describing what the PyOSG wrappers are doing: 1) This boost::python code registers the Vec3Array pyathon type: ArrayVec3.cpp: namespace PyOSG { void init_ArrayVec3() { TemplateCompArray<osg::Vec3Array, osg::Vec3> Vec3Array("Vec3Array"); } } where the referenced types are defined in the OpenSceneGraph (OSG) headers as: namespace osg { class Vec3f { float _v[3]; ... }; typedef Vec3f Vec3; // <------- Vec3 template<typename T, Array::Type ARRAYTYPE, int DataSize, int DataType> class TemplateArray : public Array, public std::vector<T> { ... } typedef TemplateArray<Vec3,Array::Vec3ArrayType,3,GL_FLOAT> Vec3Array; // <------- Vec3Array } 2) Here's the TemplateCompArray definition referred to in the boost::python code above: TemplateCompArray.hpp: using namespace boost::python; template <typename T, typename BT> class TemplateCompArray { private: class_<T, osg::ref_ptr<T>, bases<osg::Array>, boost::noncopyable> _array; public : TemplateCompArray(const char * name) : _array(name, boost::python::no_init) { _array .def(init<>()) ... .def("__getitem__", &getitem, return_internal_reference<>()) ... ; } static BT& getitem(T * self, int idx) { if (self->size() <= (unsigned int) idx) { PyErr_SetString(PyExc_IndexError, "assignment index out of range"); throw_error_already_set(); } return (*self)[idx]; } ... }; 3) Here is the b::p wrap for one OpenSceneGraph API which can return a ptr to a Vec3Array (actually, a pointer an Array, it's base class). This retrieved pointer causes problems when I try to call one of the Vec3Array's methods with it: Geometry.cpp: namespace PyOSG { void init_Geometry() { ... { class_<osg::Geometry, osg::ref_ptr<osg::Geometry>, bases<osg::Drawable> > geometry("Geometry", no_init); scope geometry_scope(geometry); geometry .def(init<>()) ... .def("getVertexArray", // <------------ THIS ONE (osg::Array *(osg::Geometry::*)()) &osg::Geometry::getVertexArray, return_value_policy<manage_osg_object>()) ... ; ... } ... } } // namespace PyOSG Here is the actual wrapped definition of the getVertexArray() method being wrapped: Geometry: namespace osg { class Geometry : public Drawable { public: ... Array* getVertexArray(); } 4) And finally here's the definition of the "manage_osg_object" return value policy reference above in the getVertexArray() b::p wrapper: held_ptr.cpp: struct manage_osg_object { template <class T> struct apply { BOOST_STATIC_CONSTANT( bool, ok = boost::is_pointer<T>::value || boost::is_reference<T>::value); typedef typename boost::mpl::if_c< ok , boost::python::to_python_indirect<T, make_osg_holder> , manage_osg_object_requires_a_pointer_or_reference_return_type<T> >::type type; }; }; struct make_osg_holder { template <class T> static PyObject* execute(T* p) { typedef osg::ref_ptr<T> smart_pointer; typedef boost::python::objects::pointer_holder<smart_pointer, T> holder_t; smart_pointer ptr(p); return boost::python::objects::make_ptr_instance<T, holder_t>::execute(ptr); } }; The puzzling thing about all this is, when I create an osg.Vec3Array inside of Python, then I can call methods (such as __getitem__, set, etc.) on that Vec3Array from inside Python with no problem. But if I obtain a reference to a Vec3Array (e.g. returned from that Geometry::getVertexArray() API above) which was generated internal to OpenSceneGraph), then I get the error message I mentioned previously. This makes me thing something isn't being correctly done inside that "manage_osg_object" return policy, but I have no clue what that is yet. Thanks, Randall
Randall Hopper <viznut@charter.net> writes: <snip way too much information>
The puzzling thing about all this is, when I create an osg.Vec3Array inside of Python, then I can call methods (such as __getitem__, set, etc.) on that Vec3Array from inside Python with no problem.
But if I obtain a reference to a Vec3Array (e.g. returned from that Geometry::getVertexArray() API above) which was generated internal to OpenSceneGraph), then I get the error message I mentioned previously.
When you pass both kinds of Python Vec3Array object to type(), do you see the same thing in both cases?
This makes me thing something isn't being correctly done inside that "manage_osg_object" return policy, but I have no clue what that is yet.
What is the return type of get_pointer(x) where x is an osg::ref_ptr<T>? -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams: |Randall Hopper <viznut@charter.net> writes: |> The puzzling thing about all this is, when I create an osg.Vec3Array inside |> of Python, then I can call methods (such as __getitem__, set, etc.) on that |> Vec3Array from inside Python with no problem. |> |> But if I obtain a reference to a Vec3Array (e.g. returned from that |> Geometry::getVertexArray() API above) which was generated internal to |> OpenSceneGraph), then I get the error message I mentioned previously. | |When you pass both kinds of Python Vec3Array object to type(), do you |see the same thing in both cases? Yes: <class 'PyOSG.osg.Vec3Array'> |> This makes me thing something isn't being correctly done inside that |> "manage_osg_object" return policy, but I have no clue what that is |> yet. | |What is the return type of get_pointer(x) where x is an |osg::ref_ptr<T>? (T *). Here's the relevent code: namespace osg { template<class T> inline T * get_pointer(osg::ref_ptr<T> const & p) { return const_cast<T *>(p.get()); } } By the way, all efforts so far to trim down the code have been too sweeping to keep the problem. So I'm still binary searching to get even get a small test program. I wish I had some understanding of what the problem is so I could focus my efforts. Could you give me your perception of what it is? Thanks, Randall
Randall Hopper <viznut@charter.net> writes:
David Abrahams: |Randall Hopper <viznut@charter.net> writes: |> The puzzling thing about all this is, when I create an osg.Vec3Array inside |> of Python, then I can call methods (such as __getitem__, set, etc.) on that |> Vec3Array from inside Python with no problem. |> |> But if I obtain a reference to a Vec3Array (e.g. returned from that |> Geometry::getVertexArray() API above) which was generated internal to |> OpenSceneGraph), then I get the error message I mentioned previously. | |When you pass both kinds of Python Vec3Array object to type(), do you |see the same thing in both cases?
Yes:
<class 'PyOSG.osg.Vec3Array'>
|> This makes me thing something isn't being correctly done inside that |> "manage_osg_object" return policy, but I have no clue what that is |> yet. | |What is the return type of get_pointer(x) where x is an |osg::ref_ptr<T>?
(T *). Here's the relevent code:
namespace osg {
template<class T> inline T * get_pointer(osg::ref_ptr<T> const & p) { return const_cast<T *>(p.get()); }
}
By the way, all efforts so far to trim down the code have been too sweeping to keep the problem. So I'm still binary searching to get even get a small test program.
I wish I had some understanding of what the problem is so I could focus my efforts. Could you give me your perception of what it is?
From
Traceback (most recent call last): ... File "osg_to_sgg.py", line 153, in setupGeode v[0].set( 0, 1, 2 ) Boost.Python.ArgumentError: Python argument types in Vec3Array.__getitem__(Vec3Array, int) did not match C++ signature: __getitem__(osg::TemplateArray<osg::Vec3f, (osg::Array::Type)10, 3, 5126>*, int) it's pretty clear that attempt to extract a osg::TemplateArray<osg::Vec3f, (osg::Array::Type)10, 3, 5126>* from the v in the Python expression v[0] failed. Why it failed is a mystery. Certainly the 2nd argument matches, though, so you could reduce this to the problem of pass v to a wrapped C++ function with the signature void f(Vec3Array*) When a Python instance of a wrapped class T is converted to a T*, Boost.Python searches through the list of instance_holders (plural, to support MI) held by the Python instance for one that contains a T. See find_instance_impl in libs/python/src/object/class.cpp In your case the search is failing where it should succeed. I would expect you to have a single instance_holder in the chain that is in fact a python::objects::pointer_holder containing a ref_ptr<Vec3Array>. The only reasons I can think of for the match to fail are: a. some problem with typeid matching. b. the ref_ptr is in fact NULL. I think you could examine the execution of find_instance_impl in a debugger and learn a lot. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams: |When a Python instance of a wrapped class T is converted to a T*, |Boost.Python searches through the list of instance_holders (plural, to |support MI) held by the Python instance for one that contains a T. |See find_instance_impl in libs/python/src/object/class.cpp | |In your case the search is failing where it should succeed. I would |expect you to have a single instance_holder in the chain that is in |fact a python::objects::pointer_holder containing a |ref_ptr<Vec3Array>. The only reasons I can think of for the match to |fail are: | | a. some problem with typeid matching. | b. the ref_ptr is in fact NULL. | |I think you could examine the execution of find_instance_impl in a |debugger and learn a lot. Soon as I get this work project squared away, I'll do that. Thanks for the insight! That's what I needed. Randall
participants (3)
-
David Abrahams -
Randall Hopper -
Roman Yakovenko