Mailman 3 April 2015 - Cplusplus-sig

subprocess fork() sometimes hangs when called from within boost::python::exec_file
by Peter Schüller May 13, 2015

May 13, 2015

Dear all, In the hexhex project on github [1] we execute a python program using boost::python::exec_file, the respective code is in src/PythonPlugin.cpp in the function PythonPlugin::runPythonMain(...). In this python program I use subprocess.call() to run a shell application (actually graphviz). Sometimes it works, sometimes it hangs using 100% CPU. It seems to depend on how much the C++ program had to do before it executed the python part. In both cases the C++ program uses threads (is that relevant wrt. fork() in python? I found some hints online that it could be relevant). I traced it down to os.fork() in subprocess.py which returns in one case and does not return at all (neither parent nor child) in the other case. If I do strace -f on the process I get the following (the SIGPROF/rt_sigreturn/clone repeats until the disk is full): [pid 18646] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f96a5fa2a50) = ? ERESTARTNOINTR (To be restarted) [pid 18646] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- [pid 18646] rt_sigreturn() = 56 [pid 18646] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f96a5fa2a50) = ? ERESTARTNOINTR (To be restarted) [pid 18646] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- [pid 18646] rt_sigreturn() = 56 [pid 18646] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f96a5fa2a50) = ? ERESTARTNOINTR (To be restarted) [pid 18646] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- [pid 18646] rt_sigreturn() = 56 [pid 18646] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f96a5fa2a50) = ? ERESTARTNOINTR (To be restarted) [pid 18646] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- If i do strace -f on the working process it looks like this: [pid 21442] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f5f4c1e5a50) = ? ERESTARTNOINTR (To be restarted) [pid 21442] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- [pid 21442] rt_sigreturn() = 56 [pid 21442] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f5f4c1e5a50) = ? ERESTARTNOINTR (To be restarted) [pid 21442] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- [pid 21442] rt_sigreturn() = 56 [pid 21442] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f5f4c1e5a50) = ? ERESTARTNOINTR (To be restarted) [pid 21442] --- SIGPROF {si_signo=SIGPROF, si_code=SI_KERNEL} --- [pid 21442] rt_sigreturn() = 56 [pid 21442] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f5f4c1e5a50) = 21448 Process 21448 attached [pid 21448] set_robust_list(0x7f5f4c1e5a60, 24 <unfinished ...> [pid 21442] write(2, "got pid", 7got pid <unfinished ...> [pid 21448] <... set_robust_list resumed> ) = 0 [pid 21442] <... write resumed> ) = 7 [pid 21442] write(2, " ", 1 <unfinished ...> [pid 21448] close(7 <unfinished ...> If I do not do strace but attach with gdb instead, the bactrace is as follows: (gdb) bt #0 0x00007fa03c9c5025 in __libc_fork () at ../nptl/sysdeps/unix/sysv/linux/x86_64/../fork.c:130 #1 0x00007fa03ccdb155 in __fork () at ../nptl/sysdeps/unix/sysv/linux/pt-fork.c:25 #2 0x00007fa03c36d6eb in posix_fork (self=<optimized out>, noargs=<optimized out>) at ../Modules/posixmodule.c:3832 #3 0x00007fa03c2f2d33 in call_function (oparg=<optimized out>, pp_stack=0x7fffbbc90ae0) at ../Python/ceval.c:4004 #4 PyEval_EvalFrameEx (f=f@entry=0x225e950, throwflag=throwflag@entry=0) at ../Python/ceval.c:2666 #5 0x00007fa03c2f517d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=18, kws=0x2693418, kwcount=0, defs=0x0, defcount=0, closure=closure@entry=0x0) at ../Python/ceval.c:3252 #6 0x00007fa03c2f2dd8 in fast_function (nk=<optimized out>, na=18, n=18, pp_stack=0x7fffbbc90d00, func=0x7fa0351e6410) at ../Python/ceval.c:4116 #7 call_function (oparg=<optimized out>, pp_stack=0x7fffbbc90d00) at ../Python/ceval.c:4041 #8 PyEval_EvalFrameEx (f=f@entry=0x2693140, throwflag=throwflag@entry=0) at ../Python/ceval.c:2666 #9 0x00007fa03c2f517d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fa0317ee530, argcount=2, kws=kws@entry=0x7fa0317ee578, kwcount=kwcount@entry=1, defs=defs@entry=0x7fa03545bce8, defcount=defcount@entry=13, closure=0x0) at ../Python/ceval.c:3252 #10 0x00007fa03c2f53e5 in function_call.15737 (func=0x7fa0351cff50, arg=0x7fa0317ee518, kw=0x7fa0318247f8) at ../Objects/funcobject.c:526 #11 0x00007fa03c2bce23 in PyObject_Call (func=func@entry=0x7fa0351cff50, arg=arg@entry=0x7fa0317ee518, kw=kw@entry=0x7fa0318247f8) at ../Objects/abstract.c:2529 #12 0x00007fa03c21030d in instancemethod_call.8988 (func=0x7fa0351cff50, arg=0x7fa0317ee518, kw=0x7fa0318247f8) at ../Objects/classobject.c:2602 #13 0x00007fa03c2bce23 in PyObject_Call (func=func@entry=0x7fa035438f00, arg=arg@entry=0x7fa0317e8e50, kw=kw@entry=0x7fa0318247f8) at ../Objects/abstract.c:2529 #14 0x00007fa03c2c148f in slot_tp_init.26761 (self=0x7fa0317f0110, args=0x7fa0317e8e50, kwds=0x7fa0318247f8) at ../Objects/typeobject.c:5692 #15 0x00007fa03c2bf4df in type_call.26491 (type=<optimized out>, args=0x7fa0317e8e50, kwds=0x7fa0318247f8) at ../Objects/typeobject.c:745 #16 0x00007fa03c2bce23 in PyObject_Call (func=func@entry=0x1d7aab0, arg=arg@entry=0x7fa0317e8e50, kw=kw@entry=0x7fa0318247f8) at ../Objects/abstract.c:2529 #17 0x00007fa03c2ef3b1 in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffbbc91300, func=0x1d7aab0) at ../Python/ceval.c:4333 #18 PyEval_EvalFrameEx (f=f@entry=0x7fa0317f2050, throwflag=throwflag@entry=0) at ../Python/ceval.c:2705 #19 0x00007fa03c2f517d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=1, kws=0x1da91e0, kwcount=1, defs=0x0, defcount=0, closure=closure@entry=0x0) at ../Python/ceval.c:3252 #20 0x00007fa03c2f2dd8 in fast_function (nk=<optimized out>, na=1, n=3, pp_stack=0x7fffbbc91520, func=0x7fa0351cfcf8) at ../Python/ceval.c:4116 #21 call_function (oparg=<optimized out>, pp_stack=0x7fffbbc91520) at ../Python/ceval.c:4041 #22 PyEval_EvalFrameEx (f=f@entry=0x1da8fc0, throwflag=throwflag@entry=0) at ../Python/ceval.c:2666 #23 0x00007fa03c2f517d in PyEval_EvalCodeEx (co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7fa03ebdd068, argcount=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, closure=0x0) at ../Python/ceval.c:3252 #24 0x00007fa03c2f5310 in function_call.15737 (func=0x7fa0351cf5f0, arg=0x7fa03ebdd050, kw=0x0) at ../Objects/funcobject.c:526 #25 0x00007fa03c2bce23 in PyObject_Call (func=func@entry=0x7fa0351cf5f0, arg=arg@entry=0x7fa03ebdd050, kw=<optimized out>) at ../Objects/abstract.c:2529 #26 0x00007fa03c27b837 in PyEval_CallObjectWithKeywords (func=func@entry=0x7fa0351cf5f0, arg=arg@entry=0x7fa03ebdd050, kw=kw@entry=0x0) at ../Python/ceval.c:3889 #27 0x00007fa03c2feb57 in PyEval_CallFunction (obj=0x7fa0351cf5f0, format=<optimized out>) at ../Python/modsupport.c:557 #28 0x00007fa03e58b94b in dlvhex::PythonPlugin::runPythonMain(std::string) () from /home/ps/_science/projects/hex/installdir_benchmark/lib/libdlvhex2-base.so.11 #29 0x0000000000421eda in main () I am using python 2.7 on ubuntu 14.04. How can I find out why it is hanging? Could it be a bug of the os.fork() implementation? Best Regards, Peter [1] https://github.com/hexhex/core/blob/master/src/PythonPlugin.cpp -- http://www.peterschueller.com/

2 7

Boost.Python As Engine's Scripting Language
by Oam April 28, 2015

April 28, 2015

(Original Message Posted At : http://www.gamedev.net/topic/667952-boostpython-as-engines-scripting-langua… <http://www.gamedev.net/topic/667952-boostpython-as-engines-scripting-langua…> ) Am I thinking about this the right way? I'm currently working with some Boost and Boost.Python in my C++ project, in attempts to get my project's library to utilize Python as it's game scripting language. Similarly I suppose to how Unity utilizes C# as one of it's game scripting languages. So let's say I have an abstract class that contains the virtual methods for overriding the specific behaviors of each particular event, such as Unity's MonoBehavior, but BaseBehavior in my case. This is what I'm trying to do and what I'm thinking the right way to do this is : 1) I have a project that is dedicated to just the library portion of the framework (Contains the BaseBehavior abstract class) class BaseBehavior() { public: virtual void OnUpdate() = 0; }; 2) Export this library portion classes / functions / etc using Boost.Python class BaseBehaviorWrap : public BaseBehavior, public boost::python::wrapper<BaseBehavior> { public: void OnUpdate() { this->get_override("OnUpdate")(); } }; BOOST_PYTHON_MODULE(ModuleName) { boost::python::class_<BaseBehaviorWrap, boost::noncopyable>("BaseBehavior") .def("OnUpdate", boost::python::pure_virtual(&BaseBehavior::OnUpdate)); } 3) Within Python, in for example: ObjectBehavior.py I import my library module inheriting from a new class from the BaseBehavior and utilizing the particular override behaviors, such as OnUpdate() import ModuleName class NewObjectBehavior(BaseBehavior): def OnUpdate(self): # Game Behavior # ... 4) Back within the C++ side I'll have a list/vector of all the objects that contain scriptable behavior and their associated attached scripts that contain this BaseBehavior script 5) I'll collect each script per object and call the [ object exec_file(str filename, object globals = object(), object locals = object()) ] function within the Boost.Python on my C++ engine side 6) From here I will utilize the boost::python::extract object to collect my particular overridden functions and call them accordingly per object I'm concerned with my fascination to go back to the C++ side to execute each file and then extract each function I need per object that contains behavior. Is this the right way to do this, or should everything be done on the Python side? I'm really trying to keep the ideal that the GameApp is utilized within C++, such as the window creation, event handling, frame calculations, etc, just the engine specific pieces. While the Python side is dedicated specifically to just Game Behaviors. -- View this message in context: http://boost.2283326.n4.nabble.com/Boost-Python-As-Engine-s-Scripting-Langu… Sent from the Python - c++-sig mailing list archive at Nabble.com.

3 3

Compile C++ to python extension
by Quentin Blanchon April 24, 2015

April 24, 2015

Hi, I'm trying to compile C++ class to a python module extension. I found a way here: http://www.boost.org/doc/libs/1_52_0/libs/python/doc/building.html#configur… but it doesn't work with versions after boost_1_54, because /tools/build/v2 doesn't exist. I need help to do this with last update and i need some explaination about how to link the needed libraries. Thanks for any help.

2 1

Passing std::wstring to Python callback
by Bogdan Cristea April 3, 2015

April 3, 2015

Hi I am using boost python for providing python bindings for a C++ library. I am using Python 2.7. My library needs to call callbacks that are defined in Python. There callbacks can take a string parameter. C++ side I am always using std::wstring. Everything works on as long as std::wstring contains only ASCII chars, but once I get something like “é” my application crashes. This issue seems to be related to the way I pass std::wstring to Python callbacks: as long as I make sure that the passed string has only ASCII chars it is ok. My problem is how do I pass unicode strings to Python. thanks Bogdan

2 1

Memory management in Boost.Python
by Ernie Lee April 2, 2015

April 2, 2015

Hello, I am working on adding Python binding to C++ project and run into the following problem: For some reason Python tries to delete objects that are still held by smart pointer instances in our C++ code. We are using std::shared_ptr for memory management and classes exposed through Boost::Python have HeldType set to std::shared_ptr<…>. This problem manifests itself in the following scenarios: - in Python i subclass C++ class, which acts as ‘callback’, and create an instance of it and pass it to C++ code, where it is held in std::shared_ptr instance. Python script itself does not use this callback object after that point. When Python script terminates i receive memory error, which seems to be triggered by attempting to double-free this callback object. - in Python i subclass C++ class, create an instance of it (call it ‘A') and store it in container-like object (call it ‘B') (B is defined in C++ and exposed to Python). B stores shared_ptr to A. As soon as Python execution point leaves current scope my object A gets deleted while B still holds shared_ptr to A. It looks to me that ether i have a bug in my wrapping code (any idea what this could be?) or in some cases Python disregards HeldType and tries to determine object lifespan by its own means (in both cases from Python point of view created object is no longer in scope so i am guessing thats why it tries to delete it). If the latter is correct, any idea how to force Python to use smart pointer class specified in HeldType? Thank you, Ernie.

2 6