[C++-sig] Re: boost::python and threads

[David Abrahams]

Vladimir Vukicevic <vladimir at pobox.com> writes:

> Howdy,
>
> I'm working on wrapping a C++ library that heavily uses threads
> internally, and makes callbacks to user-defined functions on various
> non-user-created threads.  (This is basically the situation that
> provided the impetus for PEP 311, after discussion on python-dev --

Yes, I was a prime instigator of that discussion because users were
having difficulty wrapping functions which call into QT, which in turn
may invoke PyQT callbacks, which, because they may be called on a
thread, would unconditionally acquire the GIL.  If users' functions
didn't release the GIL before calling into QT, it was very bad juju.

> note that I'm working with Python 2.3 because of this, but I beileve
> that the PEP 311 patch could be back-ported to 2.2 should anyone need
> it.) However, I'm running into problems getting the correct wrappers
> for my classes, to get the various
> Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS and
> PyGILState_Ensure/PyGILState_Release combos in the various places.
>
> A little background about my problem: from an IDL for a Foo interface,
> I generate two C++ classes, "Foo" and "FooProxy".  Both of these are
> wrapped in smart pointers and are always passed as such in the C++
> library, so we have FooHandle and FooProxyHandle.  If Foo defines a
> "bar();" operation, then class Foo will have a "virtual void bar() =
> 0;", and I have to subclass and implement it, from python or C++.  I
> have a wrapper class that does the call_method<>() dance; in this
> wrapper class, I also place PyGILState_Ensure/PyGILState_Release
> around the call_method<>() call.

Sounds right.

> FooProxy is a remote object that I get as a result of calling various
> functions to get a proxy -- it has a "void bar();" (non-virtual) that
> I can call.  

Hmm.  This is totally off-topic, but it seems odd to me (and
inconvenient, in C++) that FooProxy is not derived from Foo.

> However, because the call can be made synchronously, I
> need to wrap the call to bar() with Py_BEGIN_ALLOW_THREADS and
> Py_END_ALLOW_THREADS.  

Well, not exactly, IIUC.  The reasons to wrap it in
Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS are:

   1. Because it might take a long time and you want other threads to
      be able to use the Python interpreter.

   2. Because the call actually *depends* on other threads using the
      Python interpreter during its execution in order to be able to
      reach completion.

> It's this problem that I'm directly dealing with -- I can't figure
> out a way to wrap class FooProxy such that I can manage the current
> ThreadState and the GIL around them.  I can create another wrapper
> class 

Do you mean like a virtual-function-dispatching class, the kind with
the initial "PyObject* self" argument to its constructors?

> that does the right thing and calls FooProxy::bar(), but
> functions are returning a FooProxy (not mywrapper_for_FooProxy)

I thought you said that all the C++ interfaces passed FooProxyHandle
(?)

> and I already have a Handle<FooProxy> as a wrapper specified in the
> boost::python class_... so I end up with incorrect types half the
> time.

Specifically, what problem are you having?  Could you show a function
you are unable to wrap properly and describe the specific problems
you're having?

> However, an alternative solution that I was thinknig of is to bake the
> thread smarts directly into boost::python.  

I have always thought that should happen in some form or other anyway.

> It seems to me that call_method and similar should always be wrapped
> with a PyGILState_Ensure/PyGILState_Release.  

Some people know their extensions are only being called on the main
thread and not everyone will be ready to pay the cost for acquiring
the GIL on every callback.  IMO supplying a GILState class which
acquires on construction and releases on destruction *should* be
enough.

> Also, an additional call policy could be added, something like
> "allow_threads", which would cause b::p to call
> Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS around the function
> invocation on C++.  This seems like a much more generally useful
> solution than me trying to do the dance in wrapper classes.

I think some change to the requirements and usage of CallPolicies
would be neccessary in order to support that (see my reply to
Nikolay), but I agree with the general idea.

> Is this the right way to go?  

Probably.  I also think that some people want automatic
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS for *all* of their wrapped
functions, and that should probably be configurable, too.  You have to
be careful with that, though: once you do Py_BEGIN_ALLOW_THREADS you
can't use any Boost.Python facilities which use the Python 'C' API
(e.g. object, handle<>, ...) until you reach Py_END_ALLOW_THREADS.

> If so, I'd appreciate any pointers on implementing a new call
> policy, especially if it's possible to both specify allow_threads
> and a return_value_policy.

I think you'd better start by looking at the macro definition of
Py_BEGIN_ALLOW_THREADS et al and then at how policies are used to see
how state can be stored for the duration of the call they affect, as
the macros are wont to do.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com