[C++-sig] Re: Interest in luabind

[David Abrahams]

Daniel,

[Please try to cut out irrelevant quoting; thanks]

"Daniel Wallin" <dalwan01 at student.umu.se> writes:

>> "Daniel Wallin" <dalwan01 at student.umu.se> writes:
>>
>> > together with a pointer to the class_rep* associated
>> > with the pointee. The class_rep holds the inheritance
>> > tree, so we just compare type_info's and traverse the
>> > tree to perform the needed cast.
>>
>> I think that problem is a little more complicated than
>> you're making it out to be, and that your method ends up
>> being slower than it should be in inheritance graphs of
>> any size.  First of all, inheritance may be a DAG and you
>> have to prevent infinite loops if you're actually going
>> to support cross-casting.  Secondly Boost.Python caches
>> cast sequences so that given the most-derived type of an
>> object, you only have to search once for a conversion to
>> any other type, and after that you can do a simple
>> address calculation.  See
>> libs/python/src/object/inheritance.cpp.  This probably
>> should be better commented; the algorithms were hard to
>> figure out and I didn't write down rationale for them :(
>> On the other hand, maybe being fast isn't important in
>> this part of the code, and the cacheing should be
>> eliminated ;-)
>
> We only support upcasting, so our method isn't that slow.  

Surely you want to be able to go in both directions, though?
Surely not everyone using lua is interested in just speed
and not usability?

> Generally it's just a linked list traversal. We don't
> cache though, and caching is a good thing. :)

Yeah, probably.  It would be good to share all of that.

>> > Ok, how do you handle conversions of lvalues from c++
>> > -> python?  The to_python converter associated with a
>> > UDT does rvalue conversion and creates a new object,
>> > correct?
>>
>> Yeah.  Implicit conversion of lvalues by itself with no
>> ownership management is dangerous so you have to tell
>> Boost.Python to do it.  I'm sure you know this, though,
>> since luabind supports "parameter policies."  Bad name,
>> though: a primary reason for these is to manage return
>> values (which are not parameters).  So I wonder what
>> you're really asking?
>
> We convert lvalues to lua with no management by default. I
> don't think this is more dangerous than copying the
> objects, it's just seg faults instead of silent
> errors. 

<shiver>
Your way, mistakes by the user of the *interpreter* can
easily crash the system.  My way, only the guy/gal doing the
wrapping has to be careful:

         >>> x = X()
         >>> z = x.y
         >>> del x
         >>> z.foo()  # crash

The users of these interpreted environments have an
expectation that their interpreter won't *crash* just
because of the way they've used it.
</shiver>

> Both ways are equaly easy to make mistakes with.

Totally disagree.  Done my way, we force the guy/gal to
consider whether he really wants to do something unsafe
before he does it.  You probably think I copy objects by
default, but I don't. That was BPLv1.  In BPLv2 I issue an
error unless the user supplies a call policy.

Finally, let me point out that although we currently use
Python weak references to accomplish this I realized last
night that there's a *much* easier and more-efficient way
to do it using a special kind of smart pointer to refer to
the referenced object.

> Our policies primary reason is not to handle return
> values, but to handle conversion in both directions. For
> example, adopt() can be used to steal objects that are
> owned by the interpreter.
>
> void f(A*); def("f", &f, adopt(_1))

What, you just leak a reference here?  Or is it something
else?  I had a major client who was sure he was going to
need to leak references, but he eventually discovered that
the provided call policies could always be made to do
something more-intelligent, so I never put the
reference-leaker in the library.  I haven't had a single
request for it since, either.

> But yes, the name should indicate both
> directions..  ConversionPolicy perhaps.

Hmm, this is really very specific to calls, because it
*does* manage arguments and return values.  I really think
CallPolicy is better.  In any case I think we should
converge on this, one way or another; there will be more
languages, and you do want to be able to steal my users,
right? <wink>.  That'll be a lot easier if they see
familiar terminology ;-)

>> Oh, sure.  I don't have such a limited view of to-python
>> conversions as that.  It's perfectly possible (and often
>> desirable) to register converters which cause
>> std::vector<X> to be converted to a Python built-in list
>> of X objects.  It's the converter function itself which
>> may access the corresponding PyTypeObject (equivalent of
>> class_rep*), which it will always get through the static
>> initialization trick.
>
> Right, in our case it's also the converter that may access
> the class_rep*, and it is also possible to create
> converters which maps std::vector<X> <-> lua table. So we
> seem to be doing pretty much the same thing here. Except
> we access it with a map<..> and you do it alot faster. :)

OK.

>> >> >> > As mentioned before, lua can have multiple
>> >> >> > states, so it would be cool if the converters
>> >> >> > would be bound to the state somehow.
>> >>
>> >> >> Why?  It doesn't seem like it would be very useful
>> >> >> to have different states doing different
>> >> >> conversions.
>> >>
>> >> > It can be useful to be able to register different
>> >> > types in different states.
>>
>> >> Why?
>>
>> > Because different states might handle completely
>> > different tasks.
>>
>> Sure, but then aren't they going to handle different C++
>> types and/or be running different extension modules?  Do
>> you really want the same C++ type converted differently
>> *by the same extension module* in two states?  Sounds
>> like premature generalization to me, but I could be
>> wrong.
>
> I don't know.. It does seem reasonable to not allow
> different conversions for the same type.

Phew! ;-)

>> I'm not committed to the idea of a single registry.  In
>> fact we've been discussing a hierarchical registry system
>> where converters are searched starting with a local
>> module registry and proceeding upward to the package
>> level and finally ending with a global registry as a
>> fallback.
>
> Right, that seems reasonable.

Cool.  And let me also point out that if the module doesn't
have to collaborate with other modules, you don't even need
a registry lookup at static initialization time.  A static
data member of a class template is enough to create an area
of storage associated with a C++ type.  There's no central
registry at all in that case.  I have grave doubts about
whether it's worth special-casing the code for this, but it
might make threading easier to cope with.

>> My big problem was trying to figure out a scheme for
>> assigning match quality.  C++ uses a kind of
>> "substitutaiblity" rule for resolving partial ordering
>> which seemed like a good way to handle things.  How do
>> you do it?
>
> We just let every converter return a value indicating how
> good the match was, where 0 is perfect match and -1 is no
> match. When performing implicit conversions, every step in
> the conversions inreases the match value.
>
> Maybe I'm naive, but is there need for anything more complicated?

Well, it's the "multimethod problem": consider base and
derived class formal arguments which both match an actual
argument, or int <--> float conversions.  How much do you
increase the match value by for a particular match?

>> > Right. The requirement I was aiming to resolve was that we need a
>> > different set of parameters when doing our conversions.
>>
>> I consider that an implementation detail ;-)
>>
>> > You have your PyObject*, we have our (lua_State*, int).
>>
>> What's the int?
>
> An index to the object being converted on the lua stack.

Oh, I guess lua hasn't handed you a pointer to an object at
that point?  Well, OK.

>> We need separate registries within Boost.Python too; we
>> just don't have them, yet.  There's also a potential
>> issue with thread safety if you have modules using the
>> same registry initializing concurrently.  With a single
>> Python interpreter state, it's not an issue, since
>> extension module code is always entered on the main
>> thread until a mutex is explicitly released.  Anyway, I
>> want to discuss the whole issue of registry isolation in
>> the larger context of what's desirable for both systems
>> and their evolution into the future.
>
> Right. For luabind it seems reasonable to accept a single
> registry for every module, and perhaps global registry
> used by interacting modules as well.
>
> It doesn't seem that interesting to register different
> conversions for different states anymore. (at least not to
> me, but I could be wrong..). But if we where to increase
> the isolation of the registries, each state could just as
> well get their own registry.

Let's continue poking at the issues until we get clarity.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com