Mailman 3 May 2015 - Cplusplus-sig

shared_ptr and GIL crash in boost::python (patch)
by Václav Šmilauer Aug. 13, 2015

Aug. 13, 2015

Hi everybody, I have been affected for years by the [1] bug: when shared_ptr for an object created in python is released in c++ in multi-threaded code, Python crashes -- because the shared_ptr_deleter does not lock the GIL before calling Python functions (as the object is destroyed). The referenced bug also contains a 2-line patch; several people report that they have to hand-patch boost::python for their production code. In my case, I can work around the bug by locking the GIL around those few places where this scenario can occur (they are fortunately relatively few in my case), without recompiling boost. I would like to ask someone to look at the patch if it is safe to be applied, and do so. It's been first identified 2 years ago. If I can do something on my part, or other reporters, I am willing to do whatever is necessary to expedite this. Opinions? Cheers, Václav [1] https://svn.boost.org/trac/boost/ticket/8290

2 2

Boost.Python C++ object reference in Python: unexpected behaviour.
by Christoff Kok June 3, 2015

June 3, 2015

Hi, I am having an issue with Boost.Python with a very simple use case. I am returning a reference to an object, and it seems that my python object looses its C++ object's reference at a stage for some reason. Please see my *example* below reproducing this issue. *C++ Code:* #include <iostream> #include <vector> #include <string> #include <cmath> #include <boost/python.hpp> #include <boost/python/suite/indexing/vector_indexing_suite.hpp> class Car { public: Car(std::string name) : m_name(name) {} bool operator==(const Car &other) const { return m_name == other.m_name; } std::string GetName() { return m_name; } private: std::string m_name; }; class Factory { public: Factory(std::string name) : m_name(name) {} bool operator==(const Factory &other) const { return m_name == other.m_name && m_car_list == other.m_car_list; } Car& create_car(std::string name) { m_car_list.emplace_back(Car(name)); return m_car_list.back(); } std::string GetName() { return m_name; } std::vector<Car>& GetCarList() { return m_car_list;} private: std::string m_name; std::vector<Car> m_car_list; }; class Manufacturer { public: Manufacturer(std::string name) : m_name(name) {} bool operator==(const Manufacturer &other) const { return m_name == other.m_name && m_factory_list == other.m_factory_list; } Factory& create_factory(std::string name) { m_factory_list.emplace_back(Factory(name)); return m_factory_list.back(); } std::string GetName() { return m_name; } std::vector<Factory>& GetFactoryList() { return m_factory_list;} private: std::string m_name; std::vector<Factory> m_factory_list; }; BOOST_PYTHON_MODULE(carManufacturer) { using namespace boost::python; class_<Manufacturer>("Manufacturer", init<std::string>()) .add_property("factory_list", make_function(&Manufacturer::GetFactoryList, return_internal_reference<1>())) .add_property("name", &Manufacturer::GetName) .def("create_factory", &Manufacturer::create_factory, return_internal_reference<>()); class_<Factory>("Factory", init<std::string>()) .add_property("car_list", make_function(&Factory::GetCarList, return_internal_reference<1>())) .add_property("name", &Factory::GetName) .def("create_car", &Factory::create_car, return_internal_reference<>()); class_<Car>("Car", init<std::string>()) .add_property("name", &Car::GetName); class_<std::vector<Factory> >("FactoryList") .def(vector_indexing_suite<std::vector<Factory> >()); class_<std::vector<Car> >("Car") .def(vector_indexing_suite<std::vector<Car> >()); } *Python Code:* import sys sys.path[:0] = [r"bin\Release"] from carManufacturer import * vw = Manufacturer("VW") vw_bra_factory = vw.create_factory("Brazil Factory") beetle = vw_bra_factory.create_car("Beetle69") if vw_bra_factory is vw.factory_list[0]: print("equal.") else: print("NOT EQUAL") print("## I expected them to be the same reference..?") print("vw_bra_factory Car List size : " + str(len(vw_bra_factory.car_list))) print("Actual Car List size : " + str(len(vw.factory_list[0].car_list))) print("## This still works. Maybe the python objects differ, but refer to the same C++ object. I can live with that.") vw_sa_factory = vw.create_factory("South Africa Factory") print("vw_bra_factory Car List size : " + str(len(vw_bra_factory.car_list))) print("Actual Car List size : " + str(len(vw.factory_list[0].car_list))) print("## .. what? why? brazil py object has no cars now? I don't get it. I can't have any of that.") print("## What will happen if I create another car in the brazil factory?") combi = vw_bra_factory.create_car("Hippie van") print("vw_bra_factory Car List size : " + str(len(vw_bra_factory.car_list))) print("Actual Car List size : " + str(len(vw.factory_list[0].car_list))) print("## And another.") citi_golf = vw_bra_factory.create_car("Citi golf") print("vw_bra_factory Car List size : " + str(len(vw_bra_factory.car_list))) print("Actual Car List size : " + str(len(vw.factory_list[0].car_list))) print("## 'vw_bra_factory' must have lost its C++ reference it had to 'vw.factory_list[0]' when I created a new factory. Why?") *Python Output:* NOT EQUAL *## I expected them to be the same reference..?* vw_bra_factory Car List size : 1 Actual Car List size : 1 *## This still works. Maybe the python objects differ, but refer to the same C++ object. I can live with that.* vw_bra_factory Car List size : 0 Actual Car List size : 1 *## .. what? why? brazil py object has no cars now? I don't get it. I can't have any of that.* *## What will happen if I create another car in the brazil factory?* vw_bra_factory Car List size : 1 Actual Car List size : 1 *## And another.* vw_bra_factory Car List size : 2 Actual Car List size : 1 *## 'vw_bra_factory' must have lost its C++ reference it had to 'vw.factory_list[0]' when I created a new factory. Why?* This is just an example made to reproduce my real work's problem in a presentable way. In my real work, python crashes after I create a second "factory" and try to add a "car" to the first "factory". The crash occurs in C++ "create_car" method whan trying to access the "factory"'s "car" list. Does anyone have insight as to what the problem is? Any useful input will be greatly appreciated. Greetings, Christoff

3 7

C++ copy construction and Python object copies
by Stefan Seefeld May 31, 2015

May 31, 2015

Hello, Python's copy module allows for objects to be copied. The protocol for this will look up special method __copy__. It seems to me that this would trivially work for C++ objects providing a copy-constructor. However, the copy-constructor isn't automatically bound to __copy__. While I can certainly add that in user-code, I wonder why this isn't done by Boost.Python itself. Does anyone know the reasons for this ? Would it seem useful to add that feature ? Stefan -- ...ich hab' noch einen Koffer in Berlin...

3 3

Returning values to Python in C++ reference arguments
by Jason Addison May 26, 2015

May 26, 2015

How can results be returned in function arguments? I've include my example C++ extension and Python code below. I've tried, what I think, are some obvious approaches (Python objects, ctypes), though none have worked. It seems like this should be doable. If the python code owns, say, a c_int, I would thing it is mostly safe to pass it to c++ to be modified. This seems like it should be a common question, though I can't find it being explicitly addressed in the documentation All of the call policy stuff seems to only apply to the function result. Can the policy stuff be applied to the function arguments? If so, how would that work with my example? Thanks, Jason my_module.cpp >>>> /* clang++ -c -I/usr/include/python2.7 my_module.cpp clang++ -dynamiclib -o my_module.so -lboost_python -lpython2.7 my_module.o */ #include <boost/python.hpp> struct Foo { void bar(int &a, double &b) { a = 12; b = 6.2832; } }; using namespace boost::python; BOOST_PYTHON_MODULE(my_module) { class_<Foo>("Foo", init<>()) .def("bar", &Foo::bar) ; } <<<< byref.py >>>> from ctypes import * import my_module f = my_module.Foo() a = 0 b = 0. # py objects ''' f.bar(a, b) Traceback (most recent call last): File "byref.py", line 8, in <module> f.bar(a, b) Boost.Python.ArgumentError: Python argument types in Foo.bar(Foo, int, float) did not match C++ signature: bar(Foo {lvalue}, int {lvalue}, double {lvalue}) ''' x = c_int(0) y = c_double(0) # as is ''' f.bar(x, y) Traceback (most recent call last): File "byref.py", line 25, in <module> f.bar(x, y) Boost.Python.ArgumentError: Python argument types in Foo.bar(Foo, c_int, c_double) did not match C++ signature: bar(Foo {lvalue}, int {lvalue}, double {lvalue}) ''' # ctype.byref ''' f.bar(byref(x), byref(y)) Traceback (most recent call last): File "byref.py", line 36, in <module> f.bar(byref(x), byref(y)) Boost.Python.ArgumentError: Python argument types in Foo.bar(Foo, CArgObject, CArgObject) did not match C++ signature: bar(Foo {lvalue}, int {lvalue}, double {lvalue}) ''' # ctype.pointer ''' f.bar(pointer(x), pointer(y)) Traceback (most recent call last): File "byref.py", line 48, in <module> f.bar(pointer(x), pointer(y)) Boost.Python.ArgumentError: Python argument types in Foo.bar(Foo, LP_c_int, LP_c_double) did not match C++ signature: bar(Foo {lvalue}, int {lvalue}, double {lvalue}) ''' <<<<

4 5

undefined symbol: _ZTIN5boost6python15instance_holderE
by David Aldrich May 20, 2015

May 20, 2015

Hi I have an application that uses Boost Python. It links and runs fine on Centos 5 systems but fails on an Ubuntu 14.04 LTS system. The code that uses Boost Python is compiled using: g++ -c -Wall -m64 -DPYTHON_ENABLE -I../Kernel -I/usr/include/python2.7 -I/usr/include/ -O3 main.cpp -o _gnuRelease/main.o and linked into a static library using: g++ -o _gnuRelease/mylib -Wl,-whole-archive,-export-dynamic ../Kernel/_gnuRelease/libKernel.a -L/usr/lib/x86_64-linux-gnu -lboost_python-py27 -lpython2.7 -Wl,--no-whole-archive -ldl When the library is linked to my main program and run I see error: undefined symbol: _ZTIN5boost6python15instance_holderE ( typeinfo for boost::python::instance_holder ) Please can someone suggest how to fix this? Best regards David

2 2

subprocess fork() sometimes hangs when called from within boost::python::exec_file
by Peter Schüller May 13, 2015

May 13, 2015

Dear all, In the hexhex project on github [1] we execute a python program using boost::python::exec_file, the respective code is in src/PythonPlugin.cpp in the function PythonPlugin::runPythonMain(...). In this python program I use subprocess.call() to run a shell application (actually graphviz). Sometimes it works, sometimes it hangs using 100% CPU. It seems to depend on how much the C++ program had to do before it executed the python part. In both cases the C++ program uses threads (is that relevant wrt. fork() in python? I found some hints online that it could be relevant). I traced it down to os.fork() in subprocess.py which returns in one case and does not return at all (neither parent nor child) in the other case. If I do strace -f on the process I get the following (the SIGPROF/rt_sigreturn/clone repeats until the disk is full): [pid 18646] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f96a5fa2a50) = ? ERESTARTNOINTR (To be restarted) [pid 18646] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- [pid 18646] rt_sigreturn() = 56 [pid 18646] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f96a5fa2a50) = ? ERESTARTNOINTR (To be restarted) [pid 18646] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- [pid 18646] rt_sigreturn() = 56 [pid 18646] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f96a5fa2a50) = ? ERESTARTNOINTR (To be restarted) [pid 18646] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- [pid 18646] rt_sigreturn() = 56 [pid 18646] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f96a5fa2a50) = ? ERESTARTNOINTR (To be restarted) [pid 18646] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- If i do strace -f on the working process it looks like this: [pid 21442] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f5f4c1e5a50) = ? ERESTARTNOINTR (To be restarted) [pid 21442] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- [pid 21442] rt_sigreturn() = 56 [pid 21442] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f5f4c1e5a50) = ? ERESTARTNOINTR (To be restarted) [pid 21442] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- [pid 21442] rt_sigreturn() = 56 [pid 21442] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f5f4c1e5a50) = ? ERESTARTNOINTR (To be restarted) [pid 21442] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- [pid 21442] rt_sigreturn() = 56 [pid 21442] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f5f4c1e5a50) = 21448 Process 21448 attached [pid 21448] set_robust_list(0x7f5f4c1e5a60, 24 <unfinished ...> [pid 21442] write(2, "got pid", 7got pid <unfinished ...> [pid 21448] <... set_robust_list resumed> ) = 0 [pid 21442] <... write resumed> ) = 7 [pid 21442] write(2, " ", 1 <unfinished ...> [pid 21448] close(7 <unfinished ...> If I do not do strace but attach with gdb instead, the bactrace is as follows: (gdb) bt #0 0x00007fa03c9c5025 in __libc_fork () at ../nptl/sysdeps/unix/sysv/linux/x86_64/../fork.c:130 #1 0x00007fa03ccdb155 in __fork () at ../nptl/sysdeps/unix/sysv/linux/pt-fork.c:25 #2 0x00007fa03c36d6eb in posix_fork (self=<optimized out>, noargs=<optimized out>) at ../Modules/posixmodule.c:3832 #3 0x00007fa03c2f2d33 in call_function (oparg=<optimized out>, pp_stack=0x7fffbbc90ae0) at ../Python/ceval.c:4004 #4 PyEval_EvalFrameEx (f=f@entry=0x225e950, throwflag=throwflag@entry=0) at ../Python/ceval.c:2666 #5 0x00007fa03c2f517d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=18, kws=0x2693418, kwcount=0, defs=0x0, defcount=0, closure=closure@entry=0x0) at ../Python/ceval.c:3252 #6 0x00007fa03c2f2dd8 in fast_function (nk=<optimized out>, na=18, n=18, pp_stack=0x7fffbbc90d00, func=0x7fa0351e6410) at ../Python/ceval.c:4116 #7 call_function (oparg=<optimized out>, pp_stack=0x7fffbbc90d00) at ../Python/ceval.c:4041 #8 PyEval_EvalFrameEx (f=f@entry=0x2693140, throwflag=throwflag@entry=0) at ../Python/ceval.c:2666 #9 0x00007fa03c2f517d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fa0317ee530, argcount=2, kws=kws@entry=0x7fa0317ee578, kwcount=kwcount@entry=1, defs=defs@entry=0x7fa03545bce8, defcount=defcount@entry=13, closure=0x0) at ../Python/ceval.c:3252 #10 0x00007fa03c2f53e5 in function_call.15737 (func=0x7fa0351cff50, arg=0x7fa0317ee518, kw=0x7fa0318247f8) at ../Objects/funcobject.c:526 #11 0x00007fa03c2bce23 in PyObject_Call (func=func@entry=0x7fa0351cff50, arg=arg@entry=0x7fa0317ee518, kw=kw@entry=0x7fa0318247f8) at ../Objects/abstract.c:2529 #12 0x00007fa03c21030d in instancemethod_call.8988 (func=0x7fa0351cff50, arg=0x7fa0317ee518, kw=0x7fa0318247f8) at ../Objects/classobject.c:2602 #13 0x00007fa03c2bce23 in PyObject_Call (func=func@entry=0x7fa035438f00, arg=arg@entry=0x7fa0317e8e50, kw=kw@entry=0x7fa0318247f8) at ../Objects/abstract.c:2529 #14 0x00007fa03c2c148f in slot_tp_init.26761 (self=0x7fa0317f0110, args=0x7fa0317e8e50, kwds=0x7fa0318247f8) at ../Objects/typeobject.c:5692 #15 0x00007fa03c2bf4df in type_call.26491 (type=<optimized out>, args=0x7fa0317e8e50, kwds=0x7fa0318247f8) at ../Objects/typeobject.c:745 #16 0x00007fa03c2bce23 in PyObject_Call (func=func@entry=0x1d7aab0, arg=arg@entry=0x7fa0317e8e50, kw=kw@entry=0x7fa0318247f8) at ../Objects/abstract.c:2529 #17 0x00007fa03c2ef3b1 in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffbbc91300, func=0x1d7aab0) at ../Python/ceval.c:4333 #18 PyEval_EvalFrameEx (f=f@entry=0x7fa0317f2050, throwflag=throwflag@entry=0) at ../Python/ceval.c:2705 #19 0x00007fa03c2f517d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=1, kws=0x1da91e0, kwcount=1, defs=0x0, defcount=0, closure=closure@entry=0x0) at ../Python/ceval.c:3252 #20 0x00007fa03c2f2dd8 in fast_function (nk=<optimized out>, na=1, n=3, pp_stack=0x7fffbbc91520, func=0x7fa0351cfcf8) at ../Python/ceval.c:4116 #21 call_function (oparg=<optimized out>, pp_stack=0x7fffbbc91520) at ../Python/ceval.c:4041 #22 PyEval_EvalFrameEx (f=f@entry=0x1da8fc0, throwflag=throwflag@entry=0) at ../Python/ceval.c:2666 #23 0x00007fa03c2f517d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fa03ebdd068, argcount=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=0x0) at ../Python/ceval.c:3252 #24 0x00007fa03c2f5310 in function_call.15737 (func=0x7fa0351cf5f0, arg=0x7fa03ebdd050, kw=0x0) at ../Objects/funcobject.c:526 #25 0x00007fa03c2bce23 in PyObject_Call (func=func@entry=0x7fa0351cf5f0, arg=arg@entry=0x7fa03ebdd050, kw=<optimized out>) at ../Objects/abstract.c:2529 #26 0x00007fa03c27b837 in PyEval_CallObjectWithKeywords (func=func@entry=0x7fa0351cf5f0, arg=arg@entry=0x7fa03ebdd050, kw=kw@entry=0x0) at ../Python/ceval.c:3889 #27 0x00007fa03c2feb57 in PyEval_CallFunction (obj=0x7fa0351cf5f0, format=<optimized out>) at ../Python/modsupport.c:557 #28 0x00007fa03e58b94b in dlvhex::PythonPlugin::runPythonMain(std::string) () from /home/ps/_science/projects/hex/installdir_benchmark/lib/libdlvhex2-base.so.11 #29 0x0000000000421eda in main () I am using python 2.7 on ubuntu 14.04. How can I find out why it is hanging? Could it be a bug of the os.fork() implementation? Best Regards, Peter [1] https://github.com/hexhex/core/blob/master/src/PythonPlugin.cpp -- http://www.peterschueller.com/

2 7

compilation problem with str
by Alain Miniussi May 12, 2015

May 12, 2015

Hi, In: http://www.boost.org/doc/libs/1_58_0/libs/python/doc/tutorial/doc/html/pyth… It is explained how to automagicaly get a str conversion from the operator<<(ostream&,blog const*) operator. But, as explained in http://stackoverflow.com/questions/2828903/build-problems-when-adding-str-m… It only works if the following using self_ns::str; directive is present. Now, I do not see any mention of that issue in the documentation. Did I overlooked it ? Or is it actually missing in the doc ? (or is it a but in the lib ?). Thanks, -- --- Alain

1 0