From weyoibru at gmail.com Sat Aug 3 19:35:49 2013 From: weyoibru at gmail.com (yoibr) Date: Sat, 3 Aug 2013 17:35:49 +0000 (UTC) Subject: [C++-sig] Engineering Roles At a startup Message-ID: I am a freelance recruiter currently hiring Product Engineer,Dev-Ops engineer and Infrastructure engineer for a product startup backed by the same investors as Facebook, Google, Skype, Groupon. Location:Bangalore For more info visit:http://wehire4u.blogspot.in/ From weyoibru at gmail.com Sat Aug 3 20:57:02 2013 From: weyoibru at gmail.com (yoibr) Date: Sat, 3 Aug 2013 18:57:02 +0000 (UTC) Subject: [C++-sig] Engineering Roles At a startup Message-ID: I am a freelance recruiter currently hiring Product Engineer,Dev-Ops engineer and Infrastructure engineer for a product startup backed by the same investors as Facebook, Google, Skype, Groupon. Location:Bangalore For more info visit:http://wehire4u.blogspot.in/ From amohr at pixar.com Wed Aug 7 02:35:54 2013 From: amohr at pixar.com (Alex Mohr) Date: Tue, 06 Aug 2013 17:35:54 -0700 Subject: [C++-sig] Segfault with keyword arguments and overloads. Message-ID: <5201966A.3030102@pixar.com> I'm hitting a repeatable segfault trying to upgrade our boost version. I believe there may be a bug with overload resolution in the presence of keyword arguments. Here's a minimal example that crashes for me. // C++ #include static void f1(int a0, int a1) { } static void f2(int a0, int a1, int a2) { } BOOST_PYTHON_MODULE(kwargCrash) { boost::python::def("f", f1, (arg("a1")=2)); boost::python::def("f", f2, (arg("a2")=2)); } # Python import kwargCrash kwargCrash.f(0, a1=2) Version info: boost 1.51.0, Python 2.7.5, gcc 4.8.1. In the example we pass one positional argument and one keyword argument. This should fail to match the second overload (f2) since two positional arguments are required, and should match and invoke the first (f1). Instead, it segfaults. Here's my hypothesis as to what's happening from inspecting the boost.python code: In libs/python/src/object/function.cpp, function::call() (line 123) - We consider overloads in reverse reg order, so we try 'f2' first. - We pass the if on line 138, since n_actual is 2, f->m_nkeyword_values is 1, and min/max_arity==3. - The if on line 144 passes, since kwargs were supplied. - The fn has kwargs, so we take the 'else' on line 152. - We reach the 'build a new arg tuple...' else clause on line 167. - We make a tuple of size 3 and set the first element with the positional arg '0'. - Now we loop arg_pos = 1 to max_arity-1 (line 180). - Line 183, we set: kv = PyTuple_GET_ITEM(f->m_arg_names, arg_pos). - Line 189, we do: PyTuple_GET_ITEM(kv, 0). In this case, I believe kv is actually None, not a tuple, which gives us a bogus result, and I crash in the PyDict_GetItem call. Here's why I believe kv is actually None: In function.cpp again, function ctor (line 67) - Consider when an instance is created for 'f2'. - Line 72: keyword_offset gets set to 2 (max_arity = 3, num_keywords = 1). - m_arg_names is created with size 3 (max_arity) - Line 80: loop and set the first 2 elements (keyword_offset) to None. So the leading elements of the tuple get set to None, until we actually have keywords -- those elements get set to pairs of kwarg-name, kwarg-value. If my inspection is valid, then the indexing logic in function::call is busted. It assumes that it always gets a tuple of name/value for kv if it gets to line 189. But in this case it gets None since the argument at index 1 has no keyword. Does this sound plausible? I'm a bit surprised this was working for us previously, as the code looks like it hasn't really changed. Maybe we were getting lucky and Python changed in a way that breaks us? I.e. maybe PyTuple_GET_ITEM actually gave None if kv was None, and we never had None as a key in the passed keywords dict. Assuming my hypothesis is true, I'm not sure what the right fix here is off the top of my head. Naively it seems like checking for None in m_arg_names and rejecting the overload might work, but it feels like the right thing to do is to rework the logic used to decide whether or not a call has a satisfactory combination of positional and keyword args. Any thoughts appreciated. Alex From beamesleach at gmail.com Wed Aug 7 12:21:39 2013 From: beamesleach at gmail.com (Alex Leach) Date: Wed, 07 Aug 2013 11:21:39 +0100 Subject: [C++-sig] Segfault with keyword arguments and overloads. In-Reply-To: <5201966A.3030102@pixar.com> References: <5201966A.3030102@pixar.com> Message-ID: Hi, On Wed, 07 Aug 2013 01:35:54 +0100, Alex Mohr wrote: > I'm hitting a repeatable segfault trying to upgrade our boost version. I > believe there may be a bug with overload resolution in the presence of > keyword arguments. Here's a minimal example that crashes for me. > > // C++ > #include > > static void f1(int a0, int a1) { } > static void f2(int a0, int a1, int a2) { } > > BOOST_PYTHON_MODULE(kwargCrash) { > boost::python::def("f", f1, (arg("a1")=2)); > boost::python::def("f", f2, (arg("a2")=2)); > } I think the signature looks incorrect here. From my understanding of signatures, if you define one argument name, you should define the name for all arguments. On class methods, this includes adding an arg for 'self'. See http://www.boost.org/doc/libs/1_54_0/libs/python/doc/v2/function_doc_signature.html Also, it seems strange that you're assigning the final argument a default value, when the actual C++ function has none. I did add the corresponding default argument to the C++ function signatures, but that still seg faults, so I guess you've got default arguments in your actual code (good minimal example, though!). However, the function signature in the documentation - ie. from `help(kwargCrash)` - looks correct. Still, the following code makes the documentation more complete, and also it actually works:- #include #include #include static void f1(int a0, int a1=2) { } static void f2(int a0, int a1, int a2=2) { } BOOST_PYTHON_MODULE(kwargCrash) { boost::python::def("f", f1, (boost::python::arg("a0"), boost::python::arg("a1")=2)); boost::python::def("f", f2, (boost::python::arg("a0"), boost::python::arg("a1"), boost::python::arg("a2")=2)); } > > # Python > import kwargCrash > kwargCrash.f(0, a1=2) > > Version info: boost 1.51.0, Python 2.7.5, gcc 4.8.1. > > In the example we pass one positional argument and one keyword argument. > This should fail to match the second overload (f2) since two > positional arguments are required, and should match and invoke the first > (f1). Instead, it segfaults. > > Here's my hypothesis as to what's happening from inspecting the > boost.python code: > > In libs/python/src/object/function.cpp, function::call() (line 123) > - We consider overloads in reverse reg order, so we try 'f2' first. > - We pass the if on line 138, since n_actual is 2, > f->m_nkeyword_values is 1, and min/max_arity==3. > - The if on line 144 passes, since kwargs were supplied. > - The fn has kwargs, so we take the 'else' on line 152. > - We reach the 'build a new arg tuple...' else clause on line 167. > - We make a tuple of size 3 and set the first element with the > positional arg '0'. > - Now we loop arg_pos = 1 to max_arity-1 (line 180). > - Line 183, we set: kv = PyTuple_GET_ITEM(f->m_arg_names, arg_pos). > - Line 189, we do: PyTuple_GET_ITEM(kv, 0). > > In this case, I believe kv is actually None, not a tuple, which gives us > a bogus result, and I crash in the PyDict_GetItem call. Here's why I > believe kv is actually None: > > In function.cpp again, function ctor (line 67) > - Consider when an instance is created for 'f2'. > - Line 72: keyword_offset gets set to 2 (max_arity = 3, num_keywords > = 1). > - m_arg_names is created with size 3 (max_arity) > - Line 80: loop and set the first 2 elements (keyword_offset) to None. > > So the leading elements of the tuple get set to None, until we actually > have keywords -- those elements get set to pairs of kwarg-name, > kwarg-value. > > If my inspection is valid, then the indexing logic in function::call is > busted. It assumes that it always gets a tuple of name/value for kv if > it gets to line 189. But in this case it gets None since the argument > at index 1 has no keyword. True, the first arguments have no signature. I guess it was assumed that if you supply one argument's signature, you would supply them all. Looking at the docs for PyTuple_GET_ITEM, it is the same as PyTuple_GetItem, except that no checking is performed. So it probably returns NULL, which in turn causes PyDict_GetItem to seg fault. Here's my take, in the form of a working patch: --- /home/albl500/builds/boost/src/boost_1_53_0/libs/python/src/object/function.cpp.orig 2013-08-07 11:02:28.569236835 +0100 +++ /home/albl500/builds/boost/src/boost_1_53_0/libs/python/src/object/function.cpp 2013-08-07 11:09:09.688030081 +0100 @@ -177,17 +177,19 @@ // Grab remaining arguments by name from the keyword dictionary std::size_t n_actual_processed = n_unnamed_actual; + PyObject *kv, *k, *value; for (std::size_t arg_pos = n_unnamed_actual; arg_pos < max_arity ; ++arg_pos) { // Get the keyword[, value pair] corresponding - PyObject* kv = PyTuple_GET_ITEM(f->m_arg_names.ptr(), arg_pos); + kv = PyTuple_GET_ITEM(f->m_arg_names.ptr(), arg_pos); // If there were any keyword arguments, // look up the one we need for this // argument position - PyObject* value = n_keyword_actual - ? PyDict_GetItem(keywords, PyTuple_GET_ITEM(kv, 0)) - : 0; + k = PyTuple_GET_ITEM(kv, 0); + value = (k != NULL && n_keyword_actual) + ? PyDict_GetItem(keywords, k) + : 0; if (!value) { > > Does this sound plausible? I'm a bit surprised this was working for us > previously, as the code looks like it hasn't really changed. Maybe we > were getting lucky and Python changed in a way that breaks us? I.e. > maybe PyTuple_GET_ITEM actually gave None if kv was None, and we never > had None as a key in the passed keywords dict. I don't know of any code changes that would have introduced this.. > > Assuming my hypothesis is true, I'm not sure what the right fix here is > off the top of my head. Naively it seems like checking for None in > m_arg_names and rejecting the overload might work, but it feels like the > right thing to do is to rework the logic used to decide whether or not a > call has a satisfactory combination of positional and keyword args. > > Any thoughts appreciated. > > Alex HTH. Cheers, Alex From amohr at pixar.com Wed Aug 7 19:06:24 2013 From: amohr at pixar.com (Alex Mohr) Date: Wed, 07 Aug 2013 10:06:24 -0700 Subject: [C++-sig] Segfault with keyword arguments and overloads. In-Reply-To: References: <5201966A.3030102@pixar.com> Message-ID: <52027E90.2020701@pixar.com> Thanks for your response. I've responded to your comments in-line below. After further investigation, I believe this can be fixed by simply checking for None when we get 'kv' from f->m_arg_names while resolving keywords. If we encounter None, that means the overload doesn't accept a keyword argument in that position (so not enough positional args were passed), and we can reject the overload. If it works, I think that's a simple, targeted fix that isn't particularly risky. I'll try to work up a patch. On 8/7/2013 3:21 AM, Alex Leach wrote: > I think the signature looks incorrect here. From my understanding of > signatures, if you define one argument name, you should define the name > for all arguments. On class methods, this includes adding an arg for > 'self'. That makes sense if I was interested in the auto-docstring generation, but I'm not: in fact we don't use it. Reading the docs for boost::python::def, and boost::python::class_<>::def, it seems like trailing keyword arguments are supported: http://www.boost.org/doc/libs/1_54_0/libs/python/doc/v2/def.html#def-spec http://www.boost.org/doc/libs/1_54_0/libs/python/doc/v2/class.html#class_-spec In particular, "If the longest valid prefix of A1 contains N types and a1 holds M keywords, an initial sequence of the keywords are used for all but the first N - M arguments of each overload." Moreover, it would surprise me if boost.python demanded that if you want any argument to be a keyword argument then *all* arguments must be keyword arguments, and the code appears to be written to allow a mixture of positional and keyword arguments. > Also, it seems strange that you're assigning the final argument a > default value, when the actual C++ function has none. I did add the > corresponding default argument to the C++ function signatures, but that > still seg faults, so I guess you've got default arguments in your actual > code (good minimal example, though!). However, the function signature in > the documentation - ie. from `help(kwargCrash)` - looks correct. I could have added default args to the C++ functions, but they'd never be used and thus seemed superfluous so I left them out. And yes, thanks, providing keyword arguments for all arguments does workaround the crash, but it's not a very practical option for me, since I have at least tens of thousands of lines of boost.python wrapping code to deal with. It would be nice to just fix the crash bug and have this use-case work correctly. Alex From beamesleach at gmail.com Thu Aug 8 11:28:24 2013 From: beamesleach at gmail.com (Alex Leach) Date: Thu, 08 Aug 2013 10:28:24 +0100 Subject: [C++-sig] Segfault with keyword arguments and overloads. In-Reply-To: <52027E90.2020701@pixar.com> References: <5201966A.3030102@pixar.com> <52027E90.2020701@pixar.com> Message-ID: Hi Alex, On Wed, 07 Aug 2013 18:06:24 +0100, Alex Mohr wrote: > Thanks for your response. I've responded to your comments in-line below. > > After further investigation, I believe this can be fixed by simply > checking for None when we get 'kv' from f->m_arg_names while resolving > keywords. If we encounter None, that means the overload doesn't accept > a keyword argument in that position (so not enough positional args were > passed), and we can reject the overload. > > If it works, I think that's a simple, targeted fix that isn't > particularly risky. I'll try to work up a patch. > Sounds like you missed it, but I did add a fairly simple patch to my previous email. I used your code as a test case and it worked fine; no seg faults. I've copied it again below... I don't have commit privileges to Boost Python, but I think from your explanations, that this is a valid bug in BP. So it would probably be worthwhile filing a bug report, at http://www.boost.org/development/bugs.html Unless someone with commit privileges wants to consider merging it? My only thought on the patch regards return value optimisation. Would moving the `PyObject *` declarations out of the loop impair the compiler's ability to perform RVO? Hope that helps anyway. Kind regards, Alex --- ./boost_1_53_0/libs/python/src/object/function.cpp.orig 2013-08-07 11:02:28.569236835 +0100 +++ ./boost_1_53_0/libs/python/src/object/function.cpp 2013-08-07 11:56:10.926045853 +0100 @@ -177,16 +177,18 @@ // Grab remaining arguments by name from the keyword dictionary std::size_t n_actual_processed = n_unnamed_actual; + PyObject *kv, *k, *value; for (std::size_t arg_pos = n_unnamed_actual; arg_pos < max_arity ; ++arg_pos) { // Get the keyword[, value pair] corresponding - PyObject* kv = PyTuple_GET_ITEM(f->m_arg_names.ptr(), arg_pos); + kv = PyTuple_GET_ITEM(f->m_arg_names.ptr(), arg_pos); // If there were any keyword arguments, // look up the one we need for this // argument position - PyObject* value = n_keyword_actual - ? PyDict_GetItem(keywords, PyTuple_GET_ITEM(kv, 0)) + k = PyTuple_GET_ITEM(kv, 0); + value = (k != NULL && n_keyword_actual) + ? PyDict_GetItem(keywords, k) : 0; if (!value) From amohr at pixar.com Thu Aug 8 20:05:20 2013 From: amohr at pixar.com (Alex Mohr) Date: Thu, 08 Aug 2013 11:05:20 -0700 Subject: [C++-sig] Segfault with keyword arguments and overloads. In-Reply-To: References: <5201966A.3030102@pixar.com> <52027E90.2020701@pixar.com> Message-ID: <5203DDE0.9030909@pixar.com> Oh! I did miss your patch, sorry. Thanks! It looks to me like your patch calls PyTuple_GET_ITEM() with None as the python object (when kv is None). The python docs for PyTuple_GET_ITEM say, "Like PyTuple_GetItem(), but does no checking of its arguments." To me that means that the object you pass really needs to be a tuple. Here's the patch I'm using -- it simply checks to see if kv is None, which is true if the overload doesn't accept a keyword arg in that position, and if so it simply rejects the overload. This is working for me in the example and across our codebase and testsuite. I will file a bug as you suggest, and thanks again. Alex --- ./boost_1_53_0/libs/python/src/object/function.cpp.orig 2013-07-22 17:38:54.000000000 -0700 +++ ./boost_1_53_0/libs/python/src/object/function.cpp 2013-08-07 10:25:26.963988000 -0700 @@ -182,6 +182,16 @@ // Get the keyword[, value pair] corresponding PyObject* kv = PyTuple_GET_ITEM(f->m_arg_names.ptr(), arg_pos); + // If kv is None, this overload does not accept a + // keyword argument in this position, meaning that + // the caller did not supply enough positional + // arguments. Reject the overload. + if (kv == Py_None) { + PyErr_Clear(); + inner_args = handle<>(); + break; + } + // If there were any keyword arguments, // look up the one we need for this // argument position On 8/8/2013 2:28 AM, Alex Leach wrote: > Hi Alex, > > On Wed, 07 Aug 2013 18:06:24 +0100, Alex Mohr wrote: > >> Thanks for your response. I've responded to your comments in-line below. >> >> After further investigation, I believe this can be fixed by simply >> checking for None when we get 'kv' from f->m_arg_names while resolving >> keywords. If we encounter None, that means the overload doesn't >> accept a keyword argument in that position (so not enough positional >> args were passed), and we can reject the overload. >> >> If it works, I think that's a simple, targeted fix that isn't >> particularly risky. I'll try to work up a patch. >> > > > Sounds like you missed it, but I did add a fairly simple patch to my > previous email. I used your code as a test case and it worked fine; no > seg faults. I've copied it again below... > > I don't have commit privileges to Boost Python, but I think from your > explanations, that this is a valid bug in BP. So it would probably be > worthwhile filing a bug report, at > http://www.boost.org/development/bugs.html > > Unless someone with commit privileges wants to consider merging it? My > only thought on the patch regards return value optimisation. Would > moving the `PyObject *` declarations out of the loop impair the > compiler's ability to perform RVO? > > > Hope that helps anyway. > > Kind regards, > Alex > > > --- ./boost_1_53_0/libs/python/src/object/function.cpp.orig > 2013-08-07 11:02:28.569236835 +0100 > +++ ./boost_1_53_0/libs/python/src/object/function.cpp 2013-08-07 > 11:56:10.926045853 +0100 > @@ -177,16 +177,18 @@ > // Grab remaining arguments by name from the > keyword dictionary > std::size_t n_actual_processed = > n_unnamed_actual; > > + PyObject *kv, *k, *value; > for (std::size_t arg_pos = n_unnamed_actual; > arg_pos < max_arity ; ++arg_pos) > { > // Get the keyword[, value pair] > corresponding > - PyObject* kv = > PyTuple_GET_ITEM(f->m_arg_names.ptr(), arg_pos); > + kv = PyTuple_GET_ITEM(f->m_arg_names.ptr(), > arg_pos); > > // If there were any keyword arguments, > // look up the one we need for this > // argument position > - PyObject* value = n_keyword_actual > - ? PyDict_GetItem(keywords, > PyTuple_GET_ITEM(kv, 0)) > + k = PyTuple_GET_ITEM(kv, 0); > + value = (k != NULL && n_keyword_actual) > + ? PyDict_GetItem(keywords, k) > : 0; > > if (!value) From ellery-newcomer at utulsa.edu Fri Aug 30 23:15:12 2013 From: ellery-newcomer at utulsa.edu (Ellery Newcomer) Date: Fri, 30 Aug 2013 14:15:12 -0700 Subject: [C++-sig] double free segfaults Message-ID: Hello all. I have been trying to wrap a third party C++ library with boost::python (first time using boost::python!), and on a very simple example I seem to be getting double free segfaults, so I am here soliciting advice. Does anything stand out? In gdb, I can't seem to catch it calling any particular destructor more than once. In pure c++, the class I am wrapping does not exhibit any such segfaults on simple usage, eg { CMOOSApp *app = new CMOOSApp(); delete app; } and other classes in the library do not exhibit such segfaults when I wrap them. c++: #include #include using namespace boost::python; BOOST_PYTHON_MODULE(MOOSCore) { class_("MOOSApp"); } python: import MOOSCore thing = MOOSCore.MOOSApp() results: $ python test.py *** glibc detected *** python: double free or corruption (out): 0x0000000000f92b50 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3b8a4760e6] /lib64/libc.so.6[0x3b8a478c13] /usr/lib64/libboost_python.so.5(_ZN5boost6python15instance_holder10deallocateEP7_objectPv+0x15)[0x7f15ee57a0b5] /usr/lib64/libboost_python.so.5(+0x2a394)[0x7f15ee57a394] /usr/lib64/libpython2.6.so.1.0[0x3b9de9a715] /usr/lib64/libpython2.6.so.1.0[0x3b9de78287] /usr/lib64/libpython2.6.so.1.0(PyDict_SetItem+0xa7)[0x3b9de7acf7] /usr/lib64/libpython2.6.so.1.0(_PyModule_Clear+0x16d)[0x3b9de7c66d] /usr/lib64/libpython2.6.so.1.0(PyImport_Cleanup+0x18f)[0x3b9dee987f] /usr/lib64/libpython2.6.so.1.0(Py_Finalize+0x11b)[0x3b9def28ab] /usr/lib64/libpython2.6.so.1.0(Py_Main+0x596)[0x3b9deff2d6] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3b8a41ecdd] python[0x400649] the class CMOOSApp is defined here: https://oceanai.mit.edu/svn/moos-ivp-aro/releases/moos-ivp-13.5/MOOS_V10Beta_rc2/MOOSCore/Core/libMOOS/App/include/MOOS/libMOOS/App/MOOSApp.h my complete code can be found here on the off chance someone wants to tinker with it: https://bitbucket.org/ariovistus/moos-python -------------- next part -------------- An HTML attachment was scrubbed... URL: