Segfaults in object deallocation
Dear list, I've started using Boost.Python to wrap a 3rd party C++ library, and whilst running my unit tests, the Python interpreter segfault's during garbage collection. With explicit object deletion: $ gdb -q --args python -c 'import mylib; obj = mylib.MyObj(); del(obj)' ... *** Error in `/usr/bin/python': free(): invalid next size (fast): 0x0000000000806fd0 *** ======= Backtrace: ========= /usr/lib/libc.so.6(+0x7ab06)[0x7ffff74d3b06] /usr/lib/libc.so.6(+0x7b883)[0x7ffff74d4883] /usr/lib/libboost_python.so.1.53.0(_ZN5boost6python15instance_holder10deallocateEP7_objectPv+0x15)[0x7ffff004f555] /usr/lib/libboost_python.so.1.53.0(+0x265a1)[0x7ffff004f5a1] ... ======= Memory map: ======== ... or, leaving it to the garbage collector:- $ gdb -q --args python -c 'import mylib; obj = mylib.obj() ' ... Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7b2b0b0 in visit_decref () from /usr/lib/libpython2.7.so.1.0 (gdb) Quick question: What do I need to do to fix this? ========================== --- The wrapper class --- The wrapper class I've written is fairly simple; I haven't explicitly added a 'PyObject* _self' attribute, nor done any of the wiki recommendations[3] for managing ownership, as I don't see anywhere in the tutorial that recommends any of these methods. ------------------------------------ --- Using boost::python::wrapper --- The wrapped class (in 3rd party library) does have one virtual method (that is protected and I haven't exposed to Python) and a virtual destructor. I gather the right thing to do with virtual classes is to also inherit from 'boost::python::wrapper<Base>', but this doesn't affect the seg-fault. I guess I don't need to inherit from wrapper<Base>, as I am not exposing any virtual methods, nor using 'this->get_override(...)' anywhere in the class definition. Another very similar class is in the same source file though (virtual destructor and one unexposed virtual method), but this one has a public copy constructor. If I inherit from 'wrapper<Base>', I get a compilation error regarding an argument mismatch when Boost.Python passes the derived class to the copy constructor of the base class. Of course I can add noncopyable, to the class_<..> registration, but I'd like the exposed class to stay copyable, and I might like to use some of the wrapper template class' functionality. Should I wrap the copy constructor in the wrapper class too? Would it be possible to add this functionality to the wrapper<> template, when the class is not noncopyable? ------------------------------------ --- Calling thread-safe methods --- I don't think this is directly relevant, but the classes in question use the 3rd party library's own mutex implementations to ensure thread-safety. I've just got these working by writing a simple utility class that releases the GIL in the constructor and reacquires it in its destructor, using the C-Python API PyGILState_* functions[1]. The general tactic was described on the boost.python wiki[2], but the PyEval_(Save|Restore)Thread functions described caused the same assertion errors that I was getting earlier, with the pthreads mutex. This happened when the 3rd party library tried to lock the global mutex. Although this is mentioned in the C-Python manual, I would have found it useful if thread-safety was specifically mentioned in the tutorial or reference manual somewhere... I attach the solution I wrote to this problem, in case someone would want to add it and document it in Boost.Python somewhere. Usage: { MakeThreadsafe scope; //< GIL acquired // call function that acquires pthreads mutex. } //< GIL released when leave scope ------------------------------------ --- Garbage collection segfaul --- Any help with this would be really appreciated! I thought that registering a wrapped class with class_<...> was enough, to ensure an object's ref count was incremented and decremented properly. I guess that's the fundamentals of the problem, but am not sure the easiest / best / most efficient way of solving the problem.. Again, some specific documentation, on when and why this problem occurs and how to solve it, in the tutorial would be great! Looking forward to hearing back... Kind regards, Alex [1] - http://docs.python.org/2.7/c-api/init.html#non-python-created-threads [2] - http://wiki.python.org/moin/boost.python/HowTo#Multithreading_Support_for_my... [3] - http://wiki.python.org/moin/boost.python/HowTo#ownership_of_C.2B-.2B-_object -- Using Opera's mail client: http://www.opera.com/mail/
On 03/20/2013 10:46 AM, Alex Leach wrote:
Dear list,
I've started using Boost.Python to wrap a 3rd party C++ library, and whilst running my unit tests, the Python interpreter segfault's during garbage collection.
With explicit object deletion:
$ gdb -q --args python -c 'import mylib; obj = mylib.MyObj(); del(obj)' ... *** Error in `/usr/bin/python': free(): invalid next size (fast): 0x0000000000806fd0 *** ======= Backtrace: ========= /usr/lib/libc.so.6(+0x7ab06)[0x7ffff74d3b06] /usr/lib/libc.so.6(+0x7b883)[0x7ffff74d4883] /usr/lib/libboost_python.so.1.53.0(_ZN5boost6python15instance_holder10deallocateEP7_objectPv+0x15)[0x7ffff004f555]
/usr/lib/libboost_python.so.1.53.0(+0x265a1)[0x7ffff004f5a1] ... ======= Memory map: ======== ...
or, leaving it to the garbage collector:-
$ gdb -q --args python -c 'import mylib; obj = mylib.obj() ' ... Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7b2b0b0 in visit_decref () from /usr/lib/libpython2.7.so.1.0 (gdb)
Quick question: What do I need to do to fix this?
It looks very much like you're deleting an object twice. I'd guess that's probably because something went wrong with Boost.Python's understanding of ownership, but that's just a guess. It might be due to threading problems.
========================== --- The wrapper class ---
The wrapper class I've written is fairly simple; I haven't explicitly added a 'PyObject* _self' attribute, nor done any of the wiki recommendations[3] for managing ownership, as I don't see anywhere in the tutorial that recommends any of these methods.
If you are returning objects by pointer or [const] reference anywhere, you should double-check your use of call policies, to ensure you've told Python about ownership correctly. That'd be the first place I'd look for a problem. The tutorial is very incomplete, and should not be used to rule out more advanced features you may need when wrapping more advanced libraries (I wish I could point you at better documentation for those features, but it doesn't really exist beyond what it looks like you've already found).
------------------------------------ --- Using boost::python::wrapper ---
The wrapped class (in 3rd party library) does have one virtual method (that is protected and I haven't exposed to Python) and a virtual destructor. I gather the right thing to do with virtual classes is to also inherit from 'boost::python::wrapper<Base>', but this doesn't affect the seg-fault. I guess I don't need to inherit from wrapper<Base>, as I am not exposing any virtual methods, nor using 'this->get_override(...)' anywhere in the class definition.
If you do not need to override virtual methods in Python, there's no need at all to use wrapper. In that case, you're better off without it, as this will speed up your method calls and avoid complexity that could help hide your problem.
Another very similar class is in the same source file though (virtual destructor and one unexposed virtual method), but this one has a public copy constructor. If I inherit from 'wrapper<Base>', I get a compilation error regarding an argument mismatch when Boost.Python passes the derived class to the copy constructor of the base class. Of course I can add noncopyable, to the class_<..> registration, but I'd like the exposed class to stay copyable, and I might like to use some of the wrapper template class' functionality. Should I wrap the copy constructor in the wrapper class too? Would it be possible to add this functionality to the wrapper<> template, when the class is not noncopyable?
I don't think this could be added to the wrapper<> template in C++98, as I think it would require inheriting a constructor. It may not even be possible in C++11, because it may need to add a PyObject* argument; I don't exactly recall how constructors for wrapper<> work. In any case, it sounds like you don't need wrapper<> at all, so I'd remove it, and that should make this a non-issue.
------------------------------------ --- Calling thread-safe methods ---
I don't think this is directly relevant, but the classes in question use the 3rd party library's own mutex implementations to ensure thread-safety. I've just got these working by writing a simple utility class that releases the GIL in the constructor and reacquires it in its destructor, using the C-Python API PyGILState_* functions[1]. The general tactic was described on the boost.python wiki[2], but the PyEval_(Save|Restore)Thread functions described caused the same assertion errors that I was getting earlier, with the pthreads mutex. This happened when the 3rd party library tried to lock the global mutex.
Although this is mentioned in the C-Python manual, I would have found it useful if thread-safety was specifically mentioned in the tutorial or reference manual somewhere... I attach the solution I wrote to this problem, in case someone would want to add it and document it in Boost.Python somewhere.
http://www.boost.org/doc/libs/1_53_0/libs/python/doc/v2/faq.html#threadsuppo... There are other people on this list who know a lot more about this than I do, but my understanding has always been that it you use Boost.Python with threading on the C++ side, all bets are off, unless the Python interaction is strictly limited to one thread. Jim
On Wed, 20 Mar 2013 15:52:52 -0000, Jim Bosch <talljimbo@gmail.com> wrote:
http://www.boost.org/doc/libs/1_53_0/libs/python/doc/v2/faq.html#threadsuppo...
Thanks for that link. I checked out the tnfox library, which the FAQ refers to as having a thread-safe implementation of invoke.hpp. tnfox, whatever it is, also has an extensive indexing_suite, complete with a template for std::list containers ;) In case you're interested, it's in the BoostPatches.zip archive, at:- http://tnfox.git.sourceforge.net/git/gitweb.cgi?p=tnfox/tnfox;a=tree;f=Pytho... Cheers, Alex
On 20 Mar 2013 at 18:43, Alex Leach wrote:
Thanks for that link. I checked out the tnfox library, which the FAQ refers to as having a thread-safe implementation of invoke.hpp. tnfox, whatever it is, also has an extensive indexing_suite, complete with a template for std::list containers ;)
TnFOX is a metaprogrammed platform abstraction and GUI layer for my next-gen operating system platform, Tn (http://www.nedprod.com/Tn/). It was still compiling a few years, but I'd imagine it has a bad case of bitrot. Either way, that's all ten year old code and design now, and very much an orphan isolate in that it doesn't use Boost or recent patterns in C++. It also, rather unhelpfully given the modern world, wasn't written with embedded processors in mind and assumed exponential growth in CPU clock rates would continue much longer than it has. As it happens, I was recently employed by a major corporation on the basis of that next-gen platform design. As a side project I'm busy reimplementing parts of it in Boost and C++11 right now, so you /may/ see something written using Boost.ASIO and Boost.Graph hit Boost.Sandbox this summer with the hope of it eventually being accepted into Boost. I'm hoping, if it works out, that said side project makes possible many other useful things for C++17 and beyond.
In case you're interested, it's in the BoostPatches.zip archive, at:- http://tnfox.git.sourceforge.net/git/gitweb.cgi?p=tnfox/tnfox;a=tree;f=Pytho...
I'd imagine these are highly bitrotted by now. You may also find a discussion in this list's archives some time ago about what we ought to have in Boost.Python v3 (v4?) of use. There we discussed the general Python<=>C++ layer implementation, and how it would be nice if it could be arbitrarily configurable via invoke policies etc. Niall -- Any opinions or advice expressed here do NOT reflect those of my employer BlackBerry Inc. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
I've come across two problems with some python exporting Im trying to do. First problems is as follows. I have the following piece of python code: inv = ActiveGame.GetPlayerParty().GetInventory().GetItems() print str(len(inv)) print type(inv) inv["Thunder Core"] = 10 print str(len(inv)) for key in inv.iterkeys(): print key The problem is that the code simply hangs at print key. That is. nothing gets printed, and my program stops responding. Everything up to that point works correctly. inv is a boost::unordered_map<std::string, int> and it's exposed like follows: class_<boost::unordered_map<std::string, int> >("ItemCountMapUnordered") .def(map_indexing_suite<boost::unordered_map<std::string, int> >()) ; Inventory is wrapped like this: class InventoryComponent : public Components::Component { public: //additional methods omitted for example boost::unordered_map<std::string, int>& GetItems(); private: boost::unordered_map<std::string, int> ItemCount; } class_<Items::InventoryComponent, bases<Components::Component> >("InventoryComponent", init<Entity*>()) .def("GetItems", &Items::InventoryComponent::GetItems, return_value_policy<reference_existing_object>()) When I run the python code I get the output8<class 'ItemModule.ItemCountMapUnordered'>9 and then it hangs. Nothing is printed even though the map is definitely not empty and I was able to add an item to it. Is there something obvious I'm missing? I've tried switching it to an std::map from boost::unordered_map but the result was the same. The second problem I'm having seems directly related to trying to use a boost::unordered_map with map_indexing_suite. I wrap a map like follows: class_<boost::unordered_map<std::string, boost::shared_ptr<Character::BaseCharacter> > >("BasePartyMemberMap") .def(map_indexing_suite<boost::unordered_map<std::string, boost::shared_ptr<Character::BaseCharacter> > >()) ; however, when I attempt to access objects of this type I get the following error Error 26 error C2039: 'key_comp' : is not a member of 'boost::unordered::unordered_map<K,T>' c:\programming_libraries\cpp_libraries\boost_1_49_0\boost\python\suite\indexing\map_indexing_suite.hpp 155 in this case, switching boost::unordered_map to std::map solved the issue, but I'd prefer to keep boost::unordered_map if possible. I'd appreciate any help in fixing these issues and can provide more code if you need it. Thanks in advance
On Wed, 20 Mar 2013 15:52:52 -0000, Jim Bosch <talljimbo@gmail.com> wrote:
http://www.boost.org/doc/libs/1_53_0/libs/python/doc/v2/faq.html#threadsuppo...
There are other people on this list who know a lot more about this than I do, but my understanding has always been that it you use Boost.Python with threading on the C++ side, all bets are off, unless the Python interaction is strictly limited to one thread.
Hi again list, So I'm still working on this object de-allocation seg-fault problem and think I've pinpointed the problem... Any advice on what to do from here would be really appreciated! The class I'm trying to wrap (Environment) has the following member definition:- mutable FastMutex m_CacheMutex; The FastMutex class is implemented in the 3rd party library, and on Linux at least, is essentially just a wrapper around pthread_mutex. I wrote a stripped down version of 'Environment' yesterday, and when I commented out that FastMutex definition and all uses of it in 'Environment', then the seg-fault doesn't occur. If I uncomment only the definition, but leave its uses commented, then the seg-fault reappears. I guess the above definition must also be initialising the instance, as I can't see it being initialised anywhere else in the class... (Interestingly, the seg-fault doesn't occur if I compile with Intel's compiler instead of gcc, but that's not good enough really; valgrind will probably still complain). I noticed that the 3rd party library has a 'WRITE_MUTEX_EVENT' macro, which was a no-op, until I recompiled it with a LOG_MUTEX_EVENTS definition, so, I did that... With mutex events being logged, I can see the difference between when this FastMutex is defined and when its not: just before the seg-fault, the following call is made:- pthread_mutex_destroy(&m_Handle); where 'm_Handle' is a member variable of a friend class of 'FastMutex'. On Linux, 'm_Handle' is essentially declared as a pthread_mutex_t. So, in Environment's destructor, which I thought was a no-op, FastMutex's destructor is also being called. Now, that's a bit of a bummer. As far as I can tell (I'm still very new to C++; coming from Python), unlike how I hacked the initialiser list, I can't do any work either side of the chained base class destructor calls. ------------------------------------------- In an attempt to do that though, I did try this (didn't work):- namespace mylib { class MyEnv : public notmylib::Environment { public: MyEnv(); ~MyEnv(); // .... private: PyThreadState * _save; } } mylib::MyEnv::~MyEnv() { Py_UNBLOCK_THREADS; Environment::~Environment(); ///< compile error Py_BLOCK_THREADS; } No doubt obvious to anyone more experienced with C++, but this raises a compile error... ------------------------------------------- The next thing I thought to try and do, was to get the logic inside an exposed '__del__' function:- static void mylib::DelMyEnv(MyEnv* env) { PyThreadState * _save; _save = PyEval_SaveThread(); delete env; ////<-- crash PyEval_RestoreThread(_save); } BOOST_PYTHON_MODULE(pylib) { class_<mylib::MyEnv, noncopyable>("Env", init<>() ) .def("__del__", mylib::DelMyEnv ) ; } However, that also segfaults on the 'delete env' line, with gdb printing the message:- Error in `/usr/local/bin/python2-dbg': free(): invalid pointer: 0x000000000096fe50 *** That 'invalid pointer' address is for the 'env' pointer variable. With the power of print statements, I can see that I get into MyEnv's destructor, then Environment's destructor, but before that returns, crash.. The top of the backtrace is: #0 0x00007ffff74582c5 in raise () from /usr/lib/libc.so.6[0/9672] #1 0x00007ffff7459748 in abort () from /usr/lib/libc.so.6 #2 0x00007ffff7497f7b in __libc_message () from /usr/lib/libc.so.6 #3 0x00007ffff749db06 in malloc_printerr () from /usr/lib/libc.so.6 #4 0x00007ffff749e883 in _int_free () from /usr/lib/libc.so.6 #5 0x00007ffff6b0e377 in mylib::DelMyEnv (env=0x96fe50) at myenv.cpp:268 Again, any help here would be really appreciated! Any way I can help you to help me, please let me know!! Kind regards, Alex
On 22 Mar 2013 at 14:33, Alex Leach wrote:
So, in Environment's destructor, which I thought was a no-op, FastMutex's destructor is also being called. Now, that's a bit of a bummer. As far as I can tell (I'm still very new to C++; coming from Python), unlike how I hacked the initialiser list, I can't do any work either side of the chained base class destructor calls.
Separate C++ objects from their python wrappers. Have C++ objects held by shared_ptr. Have a custom deleter on shared_ptr invoke a virtual int_PythonIsAboutToDelete() function before it actually calls delete. You see as soon as destruction begins, all virtual functions cease to work, so you need a predestructor stage. Do your pre destruction cleanup like destroying mutexs in that virtual function. Or indeed anything which can throw exceptions, because destructors are (soon to be) noexcept. Hope that helps, Niall -- Any opinions or advice expressed here do NOT reflect those of my employer BlackBerry Inc. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
Thanks for the reply, and apologies for the delay in mine; I've been struggling with classes having protected destructors. Sorted now though.. On Sat, 23 Mar 2013 03:02:58 -0000, Niall Douglas <s_sourceforge@nedprod.com> wrote:
Separate C++ objects from their python wrappers. Have C++ objects held by shared_ptr.
Have a custom deleter on shared_ptr invoke a virtual int_PythonIsAboutToDelete() function before it actually calls delete. You see as soon as destruction begins, all virtual functions cease to work, so you need a predestructor stage.
This seems to be on the right track. I had some success when calling a Python-exposed __del__ method, with std::auto_ptr<Environment> as argument. I had found - with lots of printf calls - that the C++ destructor was being called twice. Each time, it called pthread_mutex_destroy, and being one too many times, that caused a seg-fault. Seemed like an ownership problem, and calling 'release' on std::auto_ptr<Environment>, in the exposed "__del__" method, got around that seg-fault, running without error. I was in the middle of replying with a success story, but on further testing, it seemed that this solution only worked when the library was linked against libpython3 / libboost_python3. When linking against the version 2.7 counterparts, seg-faults crept back in... I couldn't understand why... With that partial success, I thought that I should define a corresponding __new__ method, to allocate the memory for the object deleted by __del__. Although that seems okay to me in principal, I'm completely failing to write a successful implementation... Are there any boost python helpers for this? Again, any help greatly appreciated! // wrapper class namespace mylib { class IEnvironment : public notmylib::Environment { IEnvironment(const boost::python::list& envp); ~IEnvironment(void) {} // method implementing __new__ functionality? static boost::shared_ptr<IEnvironment> Create( boost::python::object type, const boost::python::list *envp ); } // custom deleter void DelEnvironment(boost::shared_ptr<IEnvironment> env); // pointer to start of environment array char** m_envp; // thread and gil state vars PyThreadState * _save; PyGILState_STATE gil_state; } // First function to be called in initialisation chain. const char* const* mylib::IEnvironment::InitEnvironment(const boost::python::list& envp) { int envc = boost::python::len(envp); pylib::m_envp = new char*[envc+1]; stl_input_iterator<char*> begin(envp) end; int i=0; while ( begin != end ) { pylib::m_envp[i++] = (*begin); begin++; } pylib::m_envp[i] = '\0'; pylib::gil_state = PyGILState_Ensure(); pylib::_save = PyEval_SaveThread(); return &pylib::m_envp[0]; } // Defines initialiser chain. The function body here runs last. mylib::IEnvironment::IEnvironment(const boost::python::list& envp) : Environment(InitIEnvironment(boost::ref(envp))) { PyEval_RestoreThread(pylib::_save); PyGILState_Release(pylib::gil_state); } // attempt at writing a __new__ method boost::shared_ptr<mylib::Environment> mylib::Environment::Create( boost::python::object cls, const boost::python::list& envp ) { PyTypeObject* cls_type = (PyTypeObject*)cls.ptr(); PyObject* cls_inst = PyType_GenericNew( cls_type, envp.ptr(), Py_None); if (PyType_Ready(cls_type) != 0) { throw error_already_set(); } // lots of trial and error here. Always in error, so far... return boost::make_shared<pylib::IEnvironment>( boost::python::extract<pylib::IEnvironment>( boost::python::object(boost::python::ptr(cls_inst)))); } BOOST_PYTHON_MODULE(foo) { boost::python::class_<mylib::Environment, boost::noncopyable, boost::shared_ptr<mylib::IEnvironment> ("Environment", init<const boost::python::list&>() ) .def("__new__", &mylib::IEnvironment::Create ) .staticmethod("__new__") //< necessary? .def("__del__", &pylib::DelEnvironment ) ; }
Do your pre destruction cleanup like destroying mutexs in that virtual function. Or indeed anything which can throw exceptions, because destructors are (soon to be) noexcept.
Hope that helps, Niall
Yep, thanks! Cheers, Alex -- Using Opera's mail client: http://www.opera.com/mail/
On Wed, 20 Mar 2013 15:54:45 -0000, Jaedyn K. Draper <jaedyn.cppsig@jaedyn.co> wrote:
Can you send the C++ code of MyObj? Kind of hard to diagnose a C++-side crash without any C++ code.
Sure. Sorry, thought previous email was getting long and it might have been an obvious fix for someone... Just writing a smaller example for here now, so apologies if it doesn't compile or repreoduce the problem exactly... General info about the classes in this file: one's for parsing command line arguments (with argc and argv); the other for parsing environment variables from the "classical C environment pointer" envp. Both are seg-faulting, but for brevity, just writing out the Environment class.. -------------------------------- -- 3rd party library classes --- // in libraryenv.hpp namespace library { class Environment { public: Environment(void); /// Constructor with the envp parameter. Environment(const char* const* envp); /// Destructor. virtual ~CNcbiEnvironment(void); /// Reset environment. /// /// Delete all cached entries, load new ones from "envp" (if not NULL). void Reset(const char* const* envp = 0); /// Get environment value by name. /// /// If environmnent value is not cached then call "Load(name)" to load /// the environmnent value. The loaded name/value pair will then be /// cached, too, after the call to "Get()". const string& Get(const string& name) const; /// Set an environment variable by name /// /// This will throw an exception if setting the variable fails void Set(const string& name, const string& value); /// Delete an environment variable by name void Unset(const string& name); protected: /// Load value of specified environment variable. virtual string Load(const string& name) const; private: // .... } } // in libraryenv.cpp extern char **environ; Environment::Environment(void) { Reset(environ); } Environment::Environment(const char* const* envp) { Reset(envp); } /// ..... ----------------------- -- My wrapper class --- // in pyenv.hpp class PyEnv : public library::Environment { public: PyEnv(void); /// Construct from a list or a dict. PyEnv(const boost::python::list& envp); PyEnv(const boost::python::dict& envp); virtual ~PyEnv(void); void ResetPyList(const boost::python::list& envp); void ResetPyDict(const boost::python::dict& envp); std::list<std::string> Filter(const std::string& prefix); std::list<std::string> Enumerate(void); boost::python::object Iterate(void); /* Need to wrap member functions locking pthreads mutex */ const std::string& Get(const std::string& name) const; void Set(const std::string& name, const std::string& value); void Unset(const std::string& name); }; // in pyenv.cpp PyEnv::PyEnv(void) : Environment() { } PyEnv::PyEnv(const boost::python::list& envp) { MakeThreadSafe scope; ResetPyList(envp) } PyEnv::PyEnv(const boost::python::dict& envp) { MakeThreadSafe scope; ResetPyDict(dict) } // other member functions.. void RegisterEnv(void) { boost::python::class_<PyEnv, boost::noncopyable> ("Environment", boost::python::init<>() ) .def("boost::python::init<const boost::python::list&>() ) .def("boost::python::init<const boost::python::dict&>() ) .def("reset", &PyEnv::ResetList ) .def("reset", &PyEnv::ResetDict ) .def("__getitem__", &PyEnv::Get ) .def("__setitem__", &PyEnv::Set ) .def("__delitem__", &PyEnv::Unset ) .def("__iter__", &PyEnv::Iterate ) .def("enumerate", &PyEnv::Enumerate ) .def("filter", &PyEnv::Filter ) ; } --------------------------------------- Hope that helps. I'm guessing that the problem might be fixed if I inherit from boost::shared_ptr, but I have no idea why I would need to do that here and not elsewhere. Or perhaps I've done something wrong. I'm sure the iterator and enumerator methods will be broken - haven't got around to testing them yet - but I don't see why destroying the object would cause segfaults on these classes.. Any clues?? Cheers, Alex
On 03/20/2013 12:23 PM, Alex Leach wrote:
Hope that helps. I'm guessing that the problem might be fixed if I inherit from boost::shared_ptr, but I have no idea why I would need to do that here and not elsewhere. Or perhaps I've done something wrong. I'm sure the iterator and enumerator methods will be broken - haven't got around to testing them yet - but I don't see why destroying the object would cause segfaults on these classes.. Any clues??
I don't think shared_ptr would help here (and you almost certainly don't want to inherit from it), though I'm curious what you're using to wrap std::list, as you clearly have methods that return std::list, and the standard library can be tricky to translate to Python due to things like iterator invalidation. That said, containers of strings are much less tricky than containers of wrapped objects. And given that your Python example that segfaults doesn't involve those methods at all, they probably aren't at fault. To me, everything thing here looks pretty straightforward, except the multithreading...which leads me to suspect that the multithreading is the problem. And that's something I'm afraid I don't know enough to help you with. Jim
Thanks for your replies! On Wed, 20 Mar 2013 16:34:00 -0000, Jim Bosch <talljimbo@gmail.com> wrote:
I don't think shared_ptr would help here (and you almost certainly don't want to inherit from it), though I'm curious what you're using to wrap std::list, as you clearly have methods that return std::list, and the standard library can be tricky to translate to Python due to things like iterator invalidation.
Heh, not yet anyway! Although I've laid out the foundations, I haven't yet implemented my attempt at a solution, nor tested it, as am still trying to get rid of these seg-faults here.. I did however write a template class for std::list, derived from indexing_suite. It was inspired by a question[1] on StackOverflow, but edited to fulfill the obligations of a class derived from indexing_suite (in boost/python/suite/indexing/indexing_suite.hpp). I've put it in a "boost_helpers" folder in my own project, but I'd be perfectly happy if someone wanted to put it in mainline boost. I attach it here, if you're interested in having it ;)
That said, containers of strings are much less tricky than containers of wrapped objects. And given that your Python example that segfaults doesn't involve those methods at all, they probably aren't at fault.
That's what I'm thinking.. I'm not calling a method or a factory function, just an object's constructor and then its destructor. The base class's destructor is a no-op, as is the one in my derived class. I've used that MakeThreadSafe object in every wrapped function where the base class' method locks the global mutex (source attached to first email in thread). But I was getting these seg-faults, before I even started playing with the GIL, as when compiled in release mode, the pthreads assertions don't raise an error and execution continued to this seg-fault.
To me, everything thing here looks pretty straightforward, except the multithreading...which leads me to suspect that the multithreading is the problem. And that's something I'm afraid I don't know enough to help you with.
Jim
-- Using Opera's mail client: http://www.opera.com/mail/
On Wed, 20 Mar 2013 16:53:55 -0000, Alex Leach <beamesleach@gmail.com> wrote:
I've put it in a "boost_helpers" folder in my own project, but I'd be perfectly happy if someone wanted to put it in mainline boost. I attach it here, if you're interested in having it ;)
Sorry, forgot to attach... Btw, the sort method isn't sorted yet, as I wanted to use the C++ STL sort algorithm, but didn't figure out how I could amend it to use Python's 'key' callback argument... Cheers, Alex
On Wed, 20 Mar 2013 16:53:55 -0000, Alex Leach <beamesleach@gmail.com> wrote:
On Wed, 20 Mar 2013 16:34:00 -0000, Jim Bosch <talljimbo@gmail.com> wrote:
[...] I'm curious what you're using to wrap std::list, as you clearly have methods that return std::list, and the standard library can be tricky to translate to Python due to things like iterator invalidation.
Heh, not yet anyway! Although I've laid out the foundations, I haven't yet implemented my attempt at a solution, nor tested it,
I've since got around to fixing the std::list wrapper I think I sent around before. Lots of errors in that version! Now, I've got it working, though; its sort and reverse functions use C++ STL algorithms instead of Python's, but that's probably a good thing; results appear to be consistent with those on a normal Python list. That said, I haven't profiled it for speed and compared performance against Python lists. I attach a complete example in the attached zip archive; just run 'python test_make_list.py' to compile an example extension, which exposes std::list<int> and std::list<std::string>. Unit-tests are then run to compare those against identical Python lists. make_list.zip contents:- ./test_make_list.py // Compiles example code and runs unit-tests ./test_make_list.cpp // example showing how to expose list<string> and list<int> ./boost_helpers/make_list.hpp // defines boost::python::list_indexing_suite ./boost_helpers/make_callback.hpp // needed for sort and reverse optional binary predicate ./util.py // helper functions for compiling with distutils A couple things to mention:- 1. make_callback.hpp isn't as tidy as make_list.hpp; it probably shouldn't even be necessary, but I needed something to help calling back into Python code, and this seemed to work. I wrote quite a nice template implementation using variadic templates, but the C++98 alternative isn't great. 2. I haven't (yet) exposed a good initialiser for the list classes, so it's not possible to do, e.g. >>> int_list = test_make_list.IntList([1, 2, 3]) Instead, I've been populating lists like this:- >>> int_list = test_make_list.IntList() >>> int_list[:] = [1, 2, 3] Obviously, that's not quite ideal. Any help or advice with making a templatised initialiser would be very welcome though! Kind regards, Alex
participants (5)
-
Alex Leach -
Jaedyn K. Draper -
Jay Riley -
Jim Bosch -
Niall Douglas