preventing a copy of the vector when using the vector_indexing_suite
Is there a way to set the policy for the vector_indexing_suite to not create a copy of the vector itself (not the internal elements) in python? I have a vector being held by the vector indexing suite defined like... class_< std::vector< smart_ptr<X> > > ( "vector_X" ) .def( vector_indexing_suite< std::vector< smart_ptr<X> > >() ) ; I then have a python extension of a c++ class, which has a method that takes a reference to std::vector< smart_ptr<X> > and populates it by calling the c++ parent class. If I do something like the following... class TestExtendedFactory( x.BasicFactory ): def __init__( self ): super( TestExtendedFactory, self ).__init__() def generate( self, vectorOfSmartPtrXs ): # first generate with our parent super( TestExtendedFactor, self ).generate( vectorOfSmartPtrXs ) ... the entries added by the call to BasicFactory.generate() show up just fine in python, but don't make it back to the original c++ caller of TestExtendedFactory.generate as it is using a copy of the vector not the original one. Thanks Scott
Scott McKay <skottmckay@gmail.com> writes:
Is there a way to set the policy for the vector_indexing_suite to not create a copy of the vector itself (not the internal elements) in python?
I have a vector being held by the vector indexing suite defined like...
class_< std::vector< smart_ptr<X> > > ( "vector_X" ) .def( vector_indexing_suite< std::vector< smart_ptr<X> > >() ) ;
I then have a python extension of a c++ class, which has a method that takes a reference to std::vector< smart_ptr<X> > and populates it by calling the c++ parent class.
If I do something like the following...
class TestExtendedFactory( x.BasicFactory ): def __init__( self ): super( TestExtendedFactory, self ).__init__()
def generate( self, vectorOfSmartPtrXs ): # first generate with our parent super( TestExtendedFactor, self ).generate( vectorOfSmartPtrXs ) ...
the entries added by the call to BasicFactory.generate() show up just fine in python, but don't make it back to the original c++ caller of TestExtendedFactory.generate as it is using a copy of the vector not the original one.
You've left out a lot of detail, but if you want to pass something from C++ to Python by reference, you need to wrap it in boost::ref( ) Otherwise, Boost.Python will take the safe route and pass it by value. HTH, -- Dave Abrahams Boost Consulting www.boost-consulting.com
You've left out a lot of detail, but if you want to pass something from C++ to Python by reference, you need to wrap it in boost::ref( ) Otherwise, Boost.Python will take the safe route and pass it by value.
Sorry - was trying to keep it to the essential details. I tried adding the boost:ref() call and that corrects the copy and provides a reference to the vector in the python code, however it fails when exiting from the python class' (TestExtendedFactory) generate method with 'TypeError: No registered converter was able to extract a C++ reference to type Factory from this Python object of type TestExtendedFactory' Do I need to register a custom converter, or is there some other call I'm missing to connect the python object with its base class' converters? Thanks for you help so far. S Here's a fuller picture: --- boost python code, largely created by Pyste --- Abstract base class for the Factory struct Factory_Wrapper: Factory, wrapper< Factory > { Factory_Wrapper(const Factory& p0): Factory(p0) {} Factory_Wrapper(): Factory() {} Factory& generate(std::vector<smart_ptr<X>>& p0) { boost::ref( p0 ); if (override f = this->get_override("generate")) return call< Factory& >( f.ptr(), p0); return Factory::generate(p0); } Factory& default_generate(std::vector<smart_ptr<X>>& p0) { return Factory::generate(p0); } }; void Export_SourceFactory() { class_< Factory_Wrapper, smart_ptr< Factory_Wrapper > >("Factory", init< >()) .def("generate", &Factory::generate, &Factory_Wrapper::default_generate, return_self<>()) .def(init< const Factory& >()) ; implicitly_convertible< smart_ptr< Factory_Wrapper >, smart_ptr< Factory > >(); } --- Basic version of the factory - implemented in C++ struct BasicFactory_Wrapper: BasicFactory, wrapper< BasicFactory > { BasicFactory_Wrapper(const BasicFactory& p0): BasicFactory(p0) {} BasicFactory_Wrapper(DocId p0, const Profile& p1): BasicFactory(p0, p1) {} Factory& generate(std::vector<smart_ptr<X>>& p0) { boost::ref( p0 ); if (override f = this->get_override("generate")) return call< Factory& >( f.ptr(), p0); return BasicFactory::generate(p0); } Factory& default_generate(std::vector<smart_ptr<X>>& p0) { return BasicFactory::generate(p0); } }; --- Class to export the basic factory void Export_BasicFactory() { class_< BasicFactory_Wrapper, smart_ptr< BasicFactory_Wrapper >
("BasicFactory", init< >()) .def("generate", (Factory& (BasicFactory::*)(std::vector<smart_ptr<X>>&) )&BasicFactory::generate, (Factory& (BasicFactory_Wrapper::*)(std::vector<smart_ptr<X>>&))&BasicFactory_Wrapper::default_generate, return_self<>()) .def(init< const BasicFactory& >()) ;
implicitly_convertible< smart_ptr< BasicFactory_Wrapper >, smart_ptr< BasicFactory > >(); implicitly_convertible< smart_ptr< BasicFactory >, smart_ptr< Factory > >(); } In the C++ code the section that calls Factory->generate holds the Factory in a smart_ptr, FWIW. The python code goes like... class TestExtendedFactory( s.BasicFactory ): "Example showing how to provide your own snippet factor for initial snippet selection" def __init__( self ): # call init of parent to setup smart pointers correctly super( TestExtendedFactory, self ).__init__() def generate( self, vectorOfSmartPtrs ): # call BasicFactory's generate super( TestExtendedFactory, self ).generate( vectorOfSmartPtrs )
Scott McKay <skottmckay@gmail.com> writes:
You've left out a lot of detail, but if you want to pass something from C++ to Python by reference, you need to wrap it in boost::ref( ) Otherwise, Boost.Python will take the safe route and pass it by value.
Sorry - was trying to keep it to the essential details.
I tried adding the boost:ref() call and that corrects the copy and provides a reference to the vector in the python code, however it fails when exiting from the python class' (TestExtendedFactory) generate method
Actually it's exiting the call<Factory&>( ... ) where we have the problem.
with
'TypeError: No registered converter was able to extract a C++ reference to type Factory from this Python object of type TestExtendedFactory'
From the code you posted it seems pretty straightforward to me. The Python generate() call is creating a Python TestExtendedFactory object, which is derived from the Python wrapper for BasicFactory, so it contains a C++ BasicFactory. Does it contain a C++ Factory object? not unless BasicFactory is derived from Factory and, from Boost.Python's POV, not unless its class_ names Factory as one of its bases<...>. You didn't show the declaration of BasicFactory, so it's hard to say much about it.
Do I need to register a custom converter, or is there some other call I'm missing to connect the python object with its base class' converters?
Thanks for you help so far.
S
Here's a fuller picture:
--- boost python code, largely created by Pyste
--- Abstract base class for the Factory
struct Factory_Wrapper: Factory, wrapper< Factory > { Factory_Wrapper(const Factory& p0): Factory(p0) {}
Factory_Wrapper(): Factory() {}
Factory& generate(std::vector<smart_ptr<X>>& p0) {
P.S. is this actually the code you compiled? All compilers I know of will choke on the appearance of ">>" above without an intervening space. -- Dave Abrahams Boost Consulting www.boost-consulting.com
The Python generate() call is creating a Python TestExtendedFactory object, which is derived from the Python wrapper for BasicFactory, so it contains a C++ BasicFactory. Does it contain a C++ Factory object? not unless BasicFactory is derived from Factory and, from Boost.Python's POV, not unless its class_ names Factory as one of its bases<...>. You didn't show the declaration of BasicFactory, so it's hard to say much about it.
A lack of bases was indeed the problem. I generated the source with a patched and updated version of Pyste that has new-style polymorphism and handles wrapped classes that have holders, and had an incorrect test that resulted in the bases not being included. Thanks! That has led me to a question on what is required in terms of implicit conversion calls to handle classes in the same hierarchy that are wrapped with wrapper<> and held by smart pointers. I can't seem to find a combination to allow derivation from both Factory and BasicFactory. If I have a python class that derives from Factory, held by a smart pointer, I need the implicit conversion defined below in Factory.cpp for it to work. If I have a python class that derives from BasicFactory (which itself derives from Factory), also held by a smart pointer, I need the implicit conversions in BasicFactory for that to work. However if I have both, a python class derived from BasicFactory will segfault. My guess is that that's because there's two converters between classes in the same hierarchy to a smart_ptr<Factory>, and it's looping in trying to resolve that. Am I doing something wrong in how I'm setting up the implicit conversions? Or should only the base class have the holder specificied, with derived classes getting it automatically (although I tried this and couldn't make it work)? As a side note the Factory and BasicFactory classes are exported in separate modules. I'm not sure that that's relevant though. Below is the boost python code for Factory/BasicFactory, the C++ for the example Runner class to show how the factory is used, and python to show the usage. I've also added some gdb output to show the loop and the initial entry into it. Apologies if this is too verbose. Thanks in advance for any help. Scott ------ Factory.cpp namespace { struct Factory_Wrapper: Factory, wrapper< Factory > { Factory_Wrapper(const Factory& p0): Factory(p0) {} Factory_Wrapper(): Factory() {} Factory& generate(SourceGroup& p0) { if (override f = this->get_override("generate")) return call< Factory& >( f.ptr(), boost::ref(p0)); return Factory::generate(boost::ref(p0)); } Factory& default_generate(SourceGroup& p0) { return Factory::generate(boost::ref(p0)); } }; }// namespace void Export_Factory() { class_< Factory_Wrapper, smart_ptr< Factory_Wrapper > >("Factory", init< >()) .def("generate", &Factory::generate, &Factory_Wrapper::default_generate, return_self<>()) .def(init< const Factory& >()) ; implicitly_convertible< smart_ptr< Factory_Wrapper >, smart_ptr< Factory > >(); implicitly_convertible< smart_ptr< Factory >, smart_ptr< Factory_Wrapper > >(); } ---- BasicFactory namespace { struct BasicFactory_Wrapper: BasicFactory, wrapper< BasicFactory > { BasicFactory_Wrapper(const BasicFactory& p0): BasicFactory(p0) {} BasicFactory_Wrapper(): BasicFactory() {} Factory& generate(SourceGroup& p0) { if (override f = this->get_override("generate")) return call< Factory& >( f.ptr(), boost::ref(p0)); return BasicFactory::generate(boost::ref(p0)); } Factory& default_generate(SourceGroup& p0) { return BasicFactory::generate(boost::ref(p0)); } }; }// namespace void Export_BasicFactory() { class_< BasicFactory_Wrapper, bases< Factory > , smart_ptr< BasicFactory_Wrapper > >("BasicFactory", init< >()) .def("generate", (Factory& (BasicFactory::*)(SourceGroup&) )&BasicFactory::generate, (Factory& (BasicFactory_Wrapper::*)(SourceGroup&))&BasicFactory_Wrapper::default_generate, return_self<>()) .def(init< const BasicFactory& >()) ; implicitly_convertible< smart_ptr< BasicFactory_Wrapper >, smart_ptr< Factory > >(); implicitly_convertible< smart_ptr< Factory >, smart_ptr< BasicFactory_Wrapper > >(); } ----- Dummy class to show how we use it: class Runner { SourceGroup _docs; smart_ptr<Factory> _factory; public: Runner() {} void setFactory( const smart_ptr<Factory> &factory ) { _factory = factory; } void run() { _factory->generate( _docs ); std::cout << "Generated: " << _docs.size() << " docs.\n"; } }; ------- Python usage example: class TestExtendedFactory( X.BasicFactory ): def __init__( self ): # call init of parent to setup smart pointers correctly super( TestExtendedFactory, self ).__init__() def generate( self, output ): super( TestExtendedFactory, self ).generate( output ) return self def main(): runner = X.Runner() myFactory = TestExtendedFactory() runner.setFactory( myFactory ) runner.run() #3564 0x0e3aaa10 in boost::python::converter::extract_rvalue<smart_ptr<Factory>
::operator() (this=0xdfe34180) at extract.hpp:175 #3565 0x0e3aa381 in boost::python::converter::implicit<smart_ptr<Factory>, smart_ptr<(anonymous namespace)::Factory_Wrapper> >::construct (obj=0xe1c25a4, data=0xdfe34234) at implicit.hpp:33 #3566 0x0e3b9f51 in boost::python::converter::rvalue_from_python_stage2 (source=0xe1c25a4, data=@0xdfe34234, converters=@0x81a3cd0) at boost/build-1.32.0/boost_1_32_0/libs/python/build/../src/converter/from_python.cpp:112 #3567 0x0e3aa8b2 in boost::python::converter::extract_rvalue<smart_ptr<(anonymous namespace)::Factory_Wrapper> >::operator() (this=0xdfe34230) at extract.hpp:175 #3568 0x0e3aa1cd in boost::python::converter::implicit<smart_ptr<(anonymous namespace)::Factory_Wrapper>, smart_ptr<Factory> >::construct (obj=0xe1c25a4, data=0xdfe342e4) at implicit.hpp:33 #3569 0x0e3b9f51 in boost::python::converter::rvalue_from_python_stage2 (source=0xe1c25a4, data=@0xdfe342e4, converters=@0x81a3be0) at boost/build-1.32.0/boost_1_32_0/libs/python/build/../src/converter/from_python.cpp:112 #3570 0x0e3aaa10 in boost::python::converter::extract_rvalue<smart_ptr<Factory> ::operator() (this=0xdfe342e0) at extract.hpp:175 #3571 0x0e3aa381 in boost::python::converter::implicit<smart_ptr<Factory>, smart_ptr<(anonymous namespace)::Factory_Wrapper> >::construct (obj=0xe1c25a4, data=0xdfe34394) at implicit.hpp:33 #3572 0x0e3b9f51 in boost::python::converter::rvalue_from_python_stage2 (source=0xe1c25a4, data=@0xdfe34394, converters=@0x81a3cd0) at boost/build-1.32.0/boost_1_32_0/libs/python/build/../src/converter/from_python.cpp:112 #3573 0x0e3aa8b2 in boost::python::converter::extract_rvalue<smart_ptr<(anonymous namespace)::Factory_Wrapper> >::operator() (this=0xdfe34390) at extract.hpp:175 #3574 0x0e3aa1cd in boost::python::converter::implicit<smart_ptr<(anonymous namespace)::Factory_Wrapper>, smart_ptr<Factory> >::construct (obj=0xe1c25a4, data=0xdfe34444) at implicit.hpp:33 #3575 0x0e3b9f51 in boost::python::converter::rvalue_from_python_stage2 (source=0xe1c25a4, data=@0xdfe34444, converters=@0x81a3be0) at boost/build-1.32.0/boost_1_32_0/libs/python/build/../src/converter/from_python.cpp:112
.... #35422 0x0e3aa8b2 in boost::python::converter::extract_rvalue<smart_ptr<(anonymous namespace)::Factory_Wrapper> >::operator() (this=0xdfffc570) at extract.hpp:175 #35423 0x0e3aa1cd in boost::python::converter::implicit<smart_ptr<(anonymous namespace)::Factory_Wrapper>, smart_ptr<Factory> >::construct (obj=0xe1c2644, data=0xdfffc630) at implicit.hpp:33 #35424 0x0e3b9a93 in boost::python::converter::arg_rvalue_from_python<smart_ptr<Factory> const&>::operator() (this=0xdfffc630) at arg_from_python.hpp:321 #35425 0x0e3b9705 in boost::python::detail::invoke<int, void (Runner::*)(smart_ptr<Factory> const&), boost::python::arg_from_python<Runner&>, boost::python::arg_from_python<smart_ptr<Factory> const&> > (f=@0x81a576c, tc=@0xdfffc648, ac0=@0xdfffc630) at invoke.hpp:94 #35426 0x0e3b9258 in boost::python::detail::caller_arity<2u>::impl<void (Runner::*)(smart_ptr<Factory> const&), boost::python::default_call_policies, boost::mpl::vector3<void, Runner&, smart_ptr<Factory> const&> >::operator() (this=0x81a576c, args_=0xe1bff8c) at caller.hpp:199 #35427 0x0e3b8d5e in boost::python::objects::caller_py_function_impl<boost::python::detail::caller<void (Runner::*)(smart_ptr<Factory> const&), boost::python::default_call_policies, boost::mpl::vector3<void, Runner&, smart_ptr<Factory> const&> > >::operator() (this=0x81a5768, args=0xe1bff8c, kw=0x0) at py_function.hpp:38 #35428 0x0e3c2df5 in boost::python::objects::py_function::operator() (this=0x81a5780, args=0xe1bff8c, kw=0x0) at py_function.hpp:139 #35429 0x0e3c189c in boost::python::objects::function::call (this=0x81a5778, args=0xe1bff8c, keywords=0x0) at boost/build-1.32.0/boost_1_32_0/libs/python/build/../src/object/function.cpp:212 #35430 0x0e3c4bae in boost::python::objects::(anonymous namespace)::bind_return::operator() (this=0xdfffc830) at boost/build-1.32.0/boost_1_32_0/libs/python/build/../src/object/function.cpp:505
2005/8/19, Scott McKay <skottmckay@gmail.com>:
implicitly_convertible< smart_ptr< Factory_Wrapper >, smart_ptr< Factory > >(); implicitly_convertible< smart_ptr< Factory >, smart_ptr< Factory_Wrapper > >(); }
Sorry about my ignorance. Why do you define those implicitly convertble pointers? [Eric Jardim]
Sorry about my ignorance. Why do you define those implicitly convertble pointers?
It's required to make smart pointers work correctly to convert between classes in the same hierarchy I believe. There may be a cleaner implicit way to make them work, but this is the approach I took after reading through old postings, faqs and various boost::python/pyste doco.
FWIW Finally had a chance to get back to this and found a way to make it work. If I remove the implicit conversion from smart_ptr<Factory> to smart_ptr<Factory_wrapper> in the base class export all is good. Not sure why that leads to the loop and seg fault if you derive from BasicFactory in python though. On 8/19/05, Scott McKay <skottmckay@gmail.com> wrote:
That has led me to a question on what is required in terms of implicit conversion calls to handle classes in the same hierarchy that are wrapped with wrapper<> and held by smart pointers. I can't seem to find a combination to allow derivation from both Factory and BasicFactory.
void Export_Factory() { class_< Factory_Wrapper, smart_ptr< Factory_Wrapper > >("Factory", init< >()) .def("generate", &Factory::generate, &Factory_Wrapper::default_generate, return_self<>()) .def(init< const Factory& >()) ;
implicitly_convertible< smart_ptr< Factory_Wrapper >, smart_ptr< Factory > >(); implicitly_convertible< smart_ptr< Factory >, smart_ptr< Factory_Wrapper > >(); }
void Export_BasicFactory() { class_< BasicFactory_Wrapper, bases< Factory > , smart_ptr< BasicFactory_Wrapper > >("BasicFactory", init< >()) .def("generate", (Factory& (BasicFactory::*)(SourceGroup&) )&BasicFactory::generate, (Factory&
(BasicFactory_Wrapper::*)(SourceGroup&))&BasicFactory_Wrapper::default_generate, return_self<>()) .def(init< const BasicFactory& >()) ;
implicitly_convertible< smart_ptr< BasicFactory_Wrapper >, smart_ptr< Factory > >(); implicitly_convertible< smart_ptr< Factory >, smart_ptr< BasicFactory_Wrapper > >(); }
participants (3)
-
David Abrahams -
Eric Jardim -
Scott McKay