[C++-sig] [Implementation] Calling wrapped functions, converters, policies

[David Abrahams]

I was recently trying to implement
http://boost-consulting.com/boost/libs/python/todo.html#injected-constructors
and ran into some issues with the way Boost.Python calls wrapped
functions.  As I began to explore boost/python/detail/caller.hpp and
boost/python/detail/invoke.hpp, I realized that the things I wanted to
do were very closely tied to some work Lijun Qin has been doing
(http://aspn.activestate.com/ASPN/Mail/Message/C++-sig/1771145), and
with the desired ability to allow converters to execute post-call
processing actions.

Since the impact of these change has the potential to be quite
sweeping, I thought we should discuss the broad requirements here
before I start implementing anything.

Here are some things I think we need:

1. Per-call state.  

   Example 1: Lijun Qin's patches decode incoming Python unicode
   object into ordinary Python strings in the Policies' precall
   function, and rely on the Policies object producing a modified
   version of the incoming Python arg tuple.

   Example 2: In implementing injected constructors I want to strip
   the incoming "self" argument before calling the inner factory
   function, then install the result in the self object.

   In both the examples above, we need to maintain a reference to the
   newly-constructed Python tuple so that we can decrement its
   reference count when the call completes.  This state is
   "per-call", since there is a new tuple created each time the
   wrapped function is called.

   Right now we can maintain per-wrapped-function state in either of
   two objects: the function itself (when it's a function object with
   an operator()), or in the Policies object.  For all practical
   purposes, though, that state has to be immutable.  Consider a
   recursive wrapped function call; the inner call will wipe out any
   information stored in the Policies object during an outer call.  

   Lijun Qin's patch gets around this by storing a std::stack of
   newly-constructed tuples in the Policies object, but not only is
   that cumbersome for the implementor, it doesn't interact well with
   multithreading.  If the wrapped function call releases the
   interpreter lock and another thread returns, the top of the
   std::stack will contain the wrong argument tuple.

   What's needed is a way to get the per-call state onto the program
   stack.  I can think of two main approaches:

   a. A copy of the Policies object is made on the stack and used
      during the function call; the copy can maintain its own state.
      An advantage of this approach is simplicity.  A disadvantage is
      that you may for things you don't use: storage for per-call
      state in the wrapped function itself, and cycles for copying
      per-call state to the stack.

   b. The policies object has a init_state() function which
      produces a new state object of a possibly-different type.  This
      doesn't have the disadvantages above, but costs complexity in
      several ways:

      i. The requirements on the Policies class become more
         complicated.  It probably needs to have a nested ::state
         type.  It's possible to get around the need for this typedef
         by passing the results of init_state() directly to a function
         template parameter (e.g. into invoke(...)), but that may
         force us to have a function call boundary at an undesirable
         place.

      ii. Policies composition may become much more complicated.  How
          do you come up with the state object corresponding to
          several composed policies?  You could use tuples... hmm,
          maybe this is a job for mpl::inherit_linearly.

   I'm leaning towards a. but I'm really not sure which one is best
   and would appreciate comments.

2. The ability to select a specialized from-Python conversion method
   for each argument.  Right now, we can only select the to-Python
   conversion method for the result, but Luabind has shown the wisdom
   of being able to do the converse, and in fact Mike Rovner was just
   asking how we could implement Luabind's "adopt" policy for
   stealing ownership of C++ objects held by auto_ptr.

   Right now, an MPL argument signature sequence gets passed to
   caller_arity<N>::impl, and the elements of that (except the first,
   which is the return type) fully determine the _static_ type (see
   below) of the from-Python argument converters.  We need a way to
   customize the type of the argument converters which are actually
   used for each argument.  I think Luabind has a solution for this.
   I can imagine several similar approaches, so I don't think this is
   hard.

3. Dynamic converter per-call state and postcall actions.  

      [Refresher: The standard from-Python converter has a static type
       that provides per-call storage for an implementation which is
       dynamically-selected based on the source Python object.  If the
       target argument type is a value or a const reference, the
       per-call storage is enough to construct an object of the target
       type, and either rvalue or lvalue converters can be used.
       Otherwise, a matching lvalue converter is required and the
       static converter type contains storage for a pointer to the
       target]

   It is sometimes desirable to allow a particular
   dynamically-selected from-python converter to use additional state.

   For example, people have asked that they be able to pass a Python
   list where a std::vector<T>& argument is expected, modify the
   vector, and have the Python list automatically updated when the
   function call returns.  Because this converter requires a
   std::vector<T> lvalue, it doesn't contain storage for the vector
   whose lifetime needs to span the length of the call.  Furthermore,
   if the call completes successfully, the source list must be updated
   to reflect the new contents of the vector.

   I propose that whichever state model is chosen in item 1 above, the
   state contains a chain of dynamically-allocated polymorphic
   postcall objects, and that dynamic converter implementations are
   passed a reference to that chain so that they can register new
   postcall actions.  The postcall actions are invoked when the call
   completes successfully, and are unconditionally deleted at the end
   of the call.  One thing I'm still not clear about is whether a
   converter's convertible() function need access to that chain.

4. The ability to release the Python interpreter lock during the
   wrapped call based on the choice of Policies.  If releasing the
   lock becomes the default behavior, it's important that it be
   automatically disabled when the wrapped function is handling
   python::object or any of its derived classes.  The release must
   happen inside the invoke(...)  function, or at least, if the
   implementation changes, after all converters have completed their
   work, since the lock must be held while any reference counts are
   changed.

   This suggests that there is an inner layer of action needed, and it
   should probably be generalized.  So the flow looks something like:

     caller<F,Policies,Sig>::operator()(
          PyObject* args, PyObject* kw)

         Create per-call state on stack

         args,kw = state.precall(args,kw)

         For each argument
             Create converter
             If not convertible, fail overload resolution

         For each converter c
             c.convert(state)  // optionally register postcall actions

         inner action (release interpreter lock)

         PyObject* result = invoke wrapped F with converted arguments

         inner "un-action" (acquire interpreter lock)

         if any postcall actions stored, execute them

         return state.postcall(result, args, kw)

    I'm not sure how this interacts with Daniel's recursive "best
    overload" resolution.


Comments?
       
-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com