From wichert at wiggy.net  Wed Feb  6 14:30:49 2013
From: wichert at wiggy.net (Wichert Akkerman)
Date: Wed, 6 Feb 2013 14:30:49 +0100
Subject: [C++-sig] GC error due to stl_input_iterator
Message-ID: <8C037086-8B89-4904-87F7-54E94CDB29C4@wiggy.net>

I am running into some unexpected errors when using stl_input_iterator. I have a fairly simple converter registered to convert an iterable list of values (custom enum) to a unordered_set. The code looks like this:

struct advicetype_set_from_python {
	advicetype_set_from_python() {
		converter::registry::push_back(&convertible, &construct, type_id<std::unordered_set<AdviceType>>());
	}

	static void* convertible(PyObject *obj) {
		if (obj==Py_None)
			return obj;

		PyObject* iter = PyObject_GetIter(obj);
		if (iter==NULL) {
			throw_error_already_set();
			return 0;
		}
		Py_DECREF(iter);
		return obj;
	}

	static void construct(PyObject* obj, converter::rvalue_from_python_stage1_data *data) {
		std::unordered_set<AdviceType> result;
		if (obj!=Py_None) {
			object o = object(handle<>(obj));
			stl_input_iterator<AdviceType> begin(o), end;
			result.insert(begin, end);
		}
		void *storage = ((converter::rvalue_from_python_storage<std::unordered_set<AdviceType>>*)data)->storage.bytes;
		new (storage) std::unordered_set<AdviceType>(result);
		data->convertible = storage;
	}
};

When using this code we were seeing segfaults due to what appears to be invalid reference counters. Since I suspected a problem in this convertor I wrote a simple test which triggers the converter a number of times:

for i in range(5000):
    my_func({AdviceType.one, AdviceType.two})

Which had an interesting result: the tests immediately aborted with a "Fatal Python error: GC object already tracked" error. Commenting out the contents of the if-statement in construct() made the problem go away, which makes we think the bug is in the stl_input_iterator used there. Am I doing something wrong here, or is this a boost bug? FWIW I am seeing this under both Boost 1.48 on an Ubuntu 12.04 machine and Boost 1.52 on OSX 10.8.

I am not sure if this is related, but clang outputs an interesting warning for the in-place new that is required here:

warning: cast from 'converter::rvalue_from_python_stage1_data *' to 'converter::rvalue_from_python_storage<std::unordered_set<AdviceType> > *' increases required alignment
      from 8 to 16 [-Wcast-align]
                        void *storage = ((converter::rvalue_from_python_storage<std::unordered_set<AdviceType>>*)data)->storage.bytes;


Wichert.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20130206/4e43ed81/attachment.html>

From wichert at wiggy.net  Thu Feb  7 09:57:17 2013
From: wichert at wiggy.net (Wichert Akkerman)
Date: Thu, 7 Feb 2013 09:57:17 +0100
Subject: [C++-sig] GC error due to stl_input_iterator
In-Reply-To: <8C037086-8B89-4904-87F7-54E94CDB29C4@wiggy.net>
References: <8C037086-8B89-4904-87F7-54E94CDB29C4@wiggy.net>
Message-ID: <2649E364-FA39-4D1D-BA26-C8C94A18A621@wiggy.net>


On Feb 6, 2013, at 14:30 , Wichert Akkerman <wichert at wiggy.net> wrote:
> Which had an interesting result: the tests immediately aborted with a "Fatal Python error: GC object already tracked" error. Commenting out the contents of the if-statement in construct() made the problem go away, which makes we think the bug is in the stl_input_iterator used there. Am I doing something wrong here, or is this a boost bug? FWIW I am seeing this under both Boost 1.48 on an Ubuntu 12.04 machine and Boost 1.52 on OSX 10.8.

I rewrote my code to use a standard PyObject_GetIter / PyIter_Next loop and that made all problems go away, which suggests that stl_input_iterator was indeed the problem.

Wichert.


From awishnick at izotope.com  Wed Feb 13 21:22:34 2013
From: awishnick at izotope.com (Aaron Wishnick)
Date: Wed, 13 Feb 2013 15:22:34 -0500
Subject: [C++-sig] [Boost.Python] Patch to add post-initialization hooks for
	value_holder
Message-ID: <CAOXNxjVZRsxqQJA+b5dcqN0DUsQbhpog=rXbaGUs=v0wZ=cvbg@mail.gmail.com>

Hello,

We are using Boost.Python to embed a Python interpreter. We have some
existing classes whose ownership semantics don't play very nicely with
being instantiated from Python, which we really want to do. It's
unfortunately not possible for us to improve this, due to a large existing
codebase; we don't want to harm our existing projects in the process of
adding Python support.

However, what we can do, is tell these instances when they have been
instantiated from within Python, and then everything will work great. We
were able to use no_init and make_constructor to use a custom constructor
that would construct the object, and then call an extra function. However,
we wanted to make this happen globally for all classes that inherit from a
base class that we have, since requiring people to remember to do this
every time a new class is exposed is likely to lead to bugs.

I was able to accomplish this by having value_holder call a function after
constructing its held object. The function,
boost_python_value_holder_post_init(), is called with a pointer to the
newly constructed object, without namespace qualification (to enable ADL),
and the default implementation takes a bool as its parameter, in order to
be the worst possible candidate in overload resolution.

This allows us to implement void
boost_python_value_holder_post_init(BaseClass*) within the same namespace
that BaseClass is defined in, and then our version of the function will get
called any time that a subclass of BaseClass is constructed from within
Boost.Python.

I'm attaching my patch below. I would really appreciate any feedback as to
whether this general concept is something Boost.Python would consider
merging in, and if so, if there's anything I can do  in terms of how my
patch is implemented in order to make it more suitable.

I'd also be happy (and interested!) to discuss any alternate ways to
accomplish what I want. To reiterate, it's really important to us that we
be able to do this for all subclasses of a certain class, automatically,
and this was the only apparent mechanism to me.

The patch follows below. Thanks for your time!

Best regards,
Aaron Wishnick

--- a/boost/python/object/value_holder.hpp
+++ b/boost/python/object/value_holder.hpp
@@ -33,6 +33,17 @@

 namespace boost { namespace python { namespace objects {

+// After a value_holder initializes its held object, this
+// function will be called with a pointer to the held object.
+// Note that it will be called with ADL, so you should define
+// it in the same namespace as that of the object you want to
+// have this called on. This function takes a bool as a catchall
+// because pointer to bool conversion is the "worst" standard
+// conversion, and any other custom overload written by a client
+// should be called instead.
+inline void boost_python_value_holder_post_init(bool) {
+}
+
 #  if BOOST_WORKAROUND(__GNUC__, == 2)
 #   define BOOST_PYTHON_UNFORWARD_LOCAL(z, n, _) BOOST_PP_COMMA_IF(n)
(typename unforward<A##n>::type)objects::do_unforward(a##n,0)
 #  else
@@ -137,6 +148,7 @@ void* value_holder_back_reference<Value,Held>::holds(
             BOOST_PP_REPEAT_1ST(N, BOOST_PYTHON_UNFORWARD_LOCAL, nil)
             )
     {
+         boost_python_value_holder_post_init(&m_held);
         python::detail::initialize_wrapper(self,
boost::addressof(this->m_held));
     }
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20130213/d4768a0a/attachment.html>

From josh.davidson at lmco.com  Sat Feb 16 23:21:23 2013
From: josh.davidson at lmco.com (Davidson, Josh)
Date: Sat, 16 Feb 2013 22:21:23 +0000
Subject: [C++-sig] Check if object is of type Boost.Python.enum
Message-ID: <86299D4CFE2C1248AA41ACF782B0D143119FB7C0@HDXDSP33.us.lmco.com>

I'm trying to find a clean way to test if an object in Python is of type Boost.Python.enum:
>>> myEnum.__bases__
(<type 'Boost.Python.enum'>,)

The easiest way would probably just be isinstance, but I can't seem to find a way to import Boost.Python.enum.  Anyone know how to import the necessary module or have a better idea?  For what I'm doing, I can't simply rely on duck typing.

Thanks,
Josh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20130216/83bddd70/attachment.html>

From matthew.scouten at gmail.com  Tue Feb 19 02:05:17 2013
From: matthew.scouten at gmail.com (matthew.scouten at gmail.com)
Date: Tue, 19 Feb 2013 01:05:17 +0000
Subject: [C++-sig] Check if object is of type Boost.Python.enum
In-Reply-To: <86299D4CFE2C1248AA41ACF782B0D143119FB7C0@HDXDSP33.us.lmco.com>
Message-ID: <em66ea8827-e566-4cd6-b13c-7830ce7f5b7d@penelope>

There is no way to import, as Boost.Python.enum or Boost.Python.class. 
This can be a problem.

This is what I used when I was doing boost.python:

def isEnumType(o):return isinstance(o, type)and 
issubclass(o,int)andnot(o isint)

This also detects other subclasses of int, so if you are doing a lot of 
that, you will also want to check that the module is your extension 
module.

http://stackoverflow.com/a/3981011/8508

------ Original Message ------
From: "Davidson, Josh" <josh.davidson at lmco.com>
To: "cplusplus-sig at python.org" <cplusplus-sig at python.org>
Sent: 2/16/2013 4:21:23 PM
Subject: [C++-sig] Check if object is of type Boost.Python.enum
>I?m trying to find a clean way to test if an object in Python is of 
>type Boost.Python.enum:
>
> >>> myEnum.__bases__
>
>(<type 'Boost.Python.enum'>,)
>
>
>
>The easiest way would probably just be isinstance, but I can?t seem to 
>find a way to import Boost.Python.enum.  Anyone know how to import the 
>necessary module or have a better idea?  For what I?m doing, I can?t 
>simply rely on duck typing.
>
>
>
>Thanks,
>
>Josh
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20130219/10313840/attachment.html>