[C++-sig] Re: Interest in luabind

David Abrahams dave at boost-consulting.com
Mon Jun 23 23:28:03 CEST 2003

"Daniel Wallin" <dalwan01 at student.umu.se> writes:

>> "Daniel Wallin" <dalwan01 at student.umu.se> writes:
>> >> > Instead of doing this we have general converters which is used
>> >> > to convert all user-defined types.
>> >> I have the same thing for most from_python conversions; the
>> >> registry is only used as a fallback in that case.
>> > Hm, doesn't the conversion of UDT's pass through the normal
>> > conversion system?
>> See get_lvalue_from_python in
>> libs/python/src/converter/from_python.cpp.  First it calls
>> find_instance_impl, which will always find a pointer to the right
>> type inside a regular wrapped class if such a pointer is findable.
>> The only thing that gets used from the registration in that case is
>> a type_info object, which has been stored in the registration, as
>> opposed to being passed as a separate parameter, just to minimize
>> the amount of object code generated in extension modules which are
>> invoking the conversion.
> Ok. This is roughly what we do too; the pointer is stored in the lua
> object

The pointer to... what? The wrapped object?

> together with a pointer to the class_rep* associated with the
> pointee. The class_rep holds the inheritance tree, so we just
> compare type_info's and traverse the tree to perform the needed
> cast.

I think that problem is a little more complicated than you're making
it out to be, and that your method ends up being slower than it should
be in inheritance graphs of any size.  First of all, inheritance may
be a DAG and you have to prevent infinite loops if you're actually
going to support cross-casting.  Secondly Boost.Python caches cast
sequences so that given the most-derived type of an object, you only
have to search once for a conversion to any other type, and after that
you can do a simple address calculation.  See
libs/python/src/object/inheritance.cpp.  This probably should be
better commented; the algorithms were hard to figure out and I didn't
write down rationale for them :( On the other hand, maybe being fast
isn't important in this part of the code, and the cacheing should be
eliminated ;-)

>> That's for from-python conversions, of course.  For to-python
>> conversions, yes, we nearly always end up consulting the
>> registration for the type.  But that's cheap, after all - there's
>> just a single method for converting any type to python so it just
>> pulls the function pointer out of the registration and invokes it.
>> Compared to the cost of constructing a Python object, an indirect
>> call gets lost in the noise.  The fact that there can only be one
>> way to do that also means that we can introduce some specializations
>> for conversion to python, which bypasses indirection for known types
>> such as int or std::string.  Note however that this bypassing
>> actually has a usability cost because the implicit conversion
>> mechanism actually consults the registration records directly, and
>> I'm not currently filling in the to-python registrations for these
>> types with specializations, so some implicit conversion sequences
>> don't work.  It may have been premature optimization to use
>> specializations here.
> Ok, how do you handle conversions of lvalues from c++ -> python? The
> to_python converter associated with a UDT does rvalue conversion and
> creates a new object, correct?

Yeah.  Implicit conversion of lvalues by itself with no ownership
management is dangerous so you have to tell Boost.Python to do it.
I'm sure you know this, though, since luabind supports "parameter
policies."  Bad name, though: a primary reason for these is to manage
return values (which are not parameters).  So I wonder what you're
really asking?

>> >> > To do this we need a map<..> lookup to find the appropriate
>> >> > converter and this really sucks.
>> >> I can't understand why you'd need that, but maybe I'm missing
>> >> something.  The general mechanism in Boost.Python is that
>> >> instance_holder::holds(type_info) will give you the address of
>> >> the contained instance if it's there.
>> > Right, we have a map<const type_info*, ..> when performing c++ ->
>> > lua conversions. You just need to do
>> > registered<T>::conversions.to_python(..); Correct?
>> Roughly speaking, yes.  But what I'm confused about is, if you're
>> using full compile-time dispatching for from-lua conversions, why
>> you don't do the same for to-lua conversions.  AFAICT, it's the
>> former where compile-time dispatch is most useful.  What's the 2nd
>> argument to the map?
> The second argument is a class_rep*, which holds information about the
> exposed type. We need this to create the holding object in lua.

Oh, sure.  I don't have such a limited view of to-python conversions
as that.  It's perfectly possible (and often desirable) to register
converters which cause std::vector<X> to be converted to a Python
built-in list of X objects.  It's the converter function itself which
may access the corresponding PyTypeObject (equivalent of class_rep*),
which it will always get through the static initialization trick.

>> >> > As mentioned before, lua can have multiple states, so it would
>> >> > be cool if the converters would be bound to the state somehow.
>> >> Why?  It doesn't seem like it would be very useful to have
>> >> different states doing different conversions.
>> > It can be useful to be able to register different types in
>> > different states.
>> Why?
> Because different states might handle completely different tasks.

Sure, but then aren't they going to handle different C++ types and/or
be running different extension modules?  Do you really want the same
C++ type converted differently *by the same extension module* in two

Sounds like premature generalization to me, but I could be wrong.

>> > Otherwise class_() would register global types and def() would
>> > register local functions. Or am I wrong in assuming that
>> > class_<T>() instantiates registered<T> and add's a few
>> > converters?
>> No, you're correct.  However, it also creates a Python type object
>> in the extension module's dictionary, just as def() creates
>> callable python objects in the module's dictionary.  I see the
>> converter registry as a separate data structure which exists in
>> parallel with the module's dictionary.  I don't see any reason to
>> have a given module register different sets of type conversions in
>> different states, even if it is going to contain different
>> types/functions (though I can't see why you'd want that either).
> As I said earlier, the states can handle different tasks. A common
> usage is object scripting in games, but you might in the same app use
> lua for parsing configuration files or scripting the GUI. It's clear
> that you don't want all these systems to have access to
> _everything_.

The question of registry isolation is a separate thing, I think.  I am
trying to suggest that a single extension module doesn't need to work
with different conversion registries in different states.

> I guess you could always register types in the global registry, and
> only expose them to the states where they are needed though, if
> there's not enough reason to use different converters for the same
> type in different states.

I'm not committed to the idea of a single registry.  In fact we've
been discussing a hierarchical registry system where converters are
searched starting with a local module registry and proceeding upward
to the package level and finally ending with a global registry as a

>> >> > Anyway, I find your converter system more appealing than
>> >> > ours. There are some issues which need to be taken care of; We
>> >> > choose best match, not first match, when trying different
>> >> > overloads. This means we need to keep the storage for the
>> >> > converter on the stack of a function that is unaware of the
>> >> > converter size (at compile time). So we need to either have a
>> >> > fixed size buffer on the stack, and hope it works, or allocate
>> >> > the storage at runtime.
>> >> I would love to have best match conversion.  I was going to do it
>> >> at one point, but realized eventually that users can sort the
>> >> overloads so that they always work so I never bothered to code
>> >> it.
>> > Do you still think best match is worth adding, or is sorting an
>> > acceptable solution?
>> I think in many cases, it's more understandable for users to be
>> able to simply control the order in which converters are tried.  It
>> certainly is *more efficient* than trying all converters, if you're
>> going to be truly compulsive about cycles, though I don't really
>> care about that.  We do have one guy, though, who's got a massively
>> confusable overload set and I think he's having trouble resolving
>> it because of the easy conversions between C++ (int, long long) and
>> Python (int, LONG).
>> http://aspn.activestate.com/ASPN/Mail/Message/1652647
>> In general, I'd prefer to have more things "just work"
>> automatically, so yeah I think it's worth adding to Boost.Python.
> Ok great.
>> >> > For clarification:
>> >> > void dispatcher(..)  { *storage here* try all overloads call best
>> >> > overload }
>> >> I've already figured out how to solve this problem; if we can
>> >> figure out how to share best-conversion technology I'll happily
>> >> code it up ;-)
>> > :) How would you do it?
>> I'll give you a hint, if you agree to cooperate on best-conversion:
> Agreed.


>> > I guess you could have static storage in the match-function and
>> > store a pointer to that in the converter data, but that wouldn't
>> > be thread safe.
>> OK, here it is, I'll tell you: you use recursion.
> Ah, I have considered that too. 

Great minds think alike ;-)

> But at the time it seemed a bit complex. You would let the 'matcher'
> functions call the next matcher, pass the current best-match value
> along and return some information that tells you if there's been a
> match further down in the recursion, and just let the matcher call
> the function when there's no better match before it on the stack.
> Something like that?

Yes, something like that.  I don't really think it's too
complicated.  My big problem was trying to figure out a scheme for
assigning match quality.  C++ uses a kind of "substitutaiblity" rule
for resolving partial ordering which seemed like a good way to handle
things.  How do you do it?

> It doesn't seem that expensive to me, the recursion won't be very
> deep anyway.

I agree; it's rare to have huge overload sets.p

>> > Perhaps we should consider parameterizing header files?
>> > namespace luabind { #define BOOST_LANG_CONVERSION_PARAMS \ (2,
>> > (lua_State*, int)) #include <blabla/conversions.hpp> }
>> Hmm, I'm not sure what you're trying to achieve here, but that kind
>> of parameterization seems unneccessary to me. we probably ought to
>> do it with templates if there's any chance at all that these
>> systems would have to be compiled together in some
>> context... though I guess with inclusion into separate namespaces
>> you could get around that.  Well OK, let's look at the requirements
>> more carefully before we jump into implementation details.  I may
>> be willing to accept additional state.
> Right. The requirement I was aiming to resolve was that we need a
> different set of parameters when doing our conversions. 

I consider that an implementation detail ;-)

> You have your PyObject*, we have our (lua_State*, int).

What's the int?

> I thought that parameterizing the implementation and including in
> different namespaces would solve all issues of that type nicely,
> though there might be far better solutions.

Maybe; I think there are lots of strategies available and which is
best probably depends on the other issues we need to address.

> Here are some notes for the conversion requirements:
>  * We need different sets of additional parameters passed through the
> conversion system. And thus we need different types of function
> pointers stored in the registry.


>  * We need to have separate registries, so that both systems can be
> used at the same time.

We need separate registries within Boost.Python too; we just don't
have them, yet.  There's also a potential issue with thread safety if
you have modules using the same registry initializing concurrently.
With a single Python interpreter state, it's not an issue, since
extension module code is always entered on the main thread until a
mutex is explicitly released.  Anyway, I want to discuss the whole
issue of registry isolation in the larger context of what's desirable
for both systems and their evolution into the future.

Dave Abrahams
Boost Consulting

More information about the Cplusplus-sig mailing list