[C++-sig] Re: Interest in luabind

[David Abrahams]

"Daniel Wallin" <dalwan01 at student.umu.se> writes:

>> >> I think that problem is a little more complicated than
>> >> you're making it out to be, and that your method ends
>> >> up being slower than it should be in inheritance
>> >> graphs of any size.  First of all, inheritance may be
>> >> a DAG and you have to prevent infinite loops if you're
>> >> actually going to support cross-casting.  Secondly
>> >> Boost.Python caches cast sequences so that given the
>> >> most-derived type of an object, you only have to
>> >> search once for a conversion to any other type, and
>> >> after that you can do a simple address calculation.
>> >> See libs/python/src/object/inheritance.cpp.  This
>> >> probably should be better commented; the algorithms
>> >> were hard to figure out and I didn't write down
>> >> rationale for them :( On the other hand, maybe being
>> >> fast isn't important in this part of the code, and the
>> >> cacheing should be eliminated ;-)
>> >
>> > We only support upcasting, so our method isn't that
>> > slow.
>>
>> Surely you want to be able to go in both directions,
>> though?  Surely not everyone using lua is interested in
>> just speed and not usability?
>
> We probably would like to be able to go in both
> directions.  We also don't want to force the user to
> compile with RTTI turned on

I'm sure that's a capability some Boost.Python users would
appreciate, too.

> so we currently supply a LUABIND_TYPEID macro to overload
> the typeid calls for a unique id for the type. 

This functions something like a specialization?

> This of course causes some problems if we want to
> downcast

And if you want to support CBD (component based development,
i.e. cross-module conversions), unless you can somehow get
all authors to agree on how to identify types.  Well, I
guess they could use strings and manually supply what
std::type_info::name() does.

> so we would need to be able to turn downcasting off, or
> let the user supply their own RTTI-system somehow.

That's pretty straightforward, fortunately.

We need to be careful about what kinds of reconfigurability
is available through macros.  Extensions linked to the same
shared library all share a link symbol space and thus are
subject to ODR problems.

>> >> > Ok, how do you handle conversions of lvalues from c++
>> >> > -> python?  The to_python converter associated with a
>> >> > UDT does rvalue conversion and creates a new object,
>> >> > correct?
>> >>
>> >> Yeah.  Implicit conversion of lvalues by itself with
>> >> no ownership management is dangerous so you have to
>> >> tell Boost.Python to do it.  I'm sure you know this,
>> >> though, since luabind supports "parameter policies."
>> >> Bad name, though: a primary reason for these is to
>> >> manage return values (which are not parameters).  So I
>> >> wonder what you're really asking?
>> >
>> > We convert lvalues to lua with no management by
>> > default. I don't think this is more dangerous than
>> > copying the objects, it's just seg faults instead of
>> > silent errors.
>>
>> <shiver>
>> Your way, mistakes by the user of the *interpreter* can
>> easily crash the system.  My way, only the guy/gal doing
>> the wrapping has to be careful:
>>
>>          >>> x = X()
>>          >>> z = x.y
>>          >>> del x
>>          >>> z.foo()  # crash
>>
>> The users of these interpreted environments have an
>> expectation that their interpreter won't *crash* just
>> because of the way they've used it.
>> </shiver>
>
> Right, I thought you always copied the object. My mistake.

OK, sorry for the heat.

>> > Both ways are equaly easy to make mistakes with.
>>
>> Totally disagree.  Done my way, we force the guy/gal to
>> consider whether he really wants to do something unsafe
>> before he does it.  You probably think I copy objects by
>> default, but I don't. That was BPLv1.  In BPLv2 I issue an
>> error unless the user supplies a call policy.
>>
>> Finally, let me point out that although we currently use
>> Python weak references to accomplish this I realized last
>> night that there's a *much* easier and more-efficient way
>> to do it using a special kind of smart pointer to refer to
>> the referenced object.
>
> Ah ok, function which returns lvalues causes compile time
> errors. 

Relatively pretty ones, too.  See
boost/python/default_call_policies.hpp

> When we decided to do it our way we thought returning
> unmanaged lvalue's would be the most common usage.  We
> only considered copying the object as an alternative,
> perhaps it's better to give compile time errors.

It's *miles* better.  Otherwise users will just blindly wrap
these things unsafely without considering the consequences.
Remember that the segfault may not occur in their tests, due
to (un)lucky usage patterns.

>> > Our policies primary reason is not to handle return
>> > values, but to handle conversion in both
>> > directions. For example, adopt() can be used to steal
>> > objects that are owned by the interpreter.
>> >
>> > void f(A*); def("f", &f, adopt(_1))
>>
>> What, you just leak a reference here?  Or is it something
>> else?  I had a major client who was sure he was going to
>> need to leak references, but he eventually discovered
>> that the provided call policies could always be made to
>> do something more-intelligent, so I never put the
>> reference-leaker in the library.  I haven't had a single
>> request for it since, either.
>
> The above is (almost) the equivalent of:
> void f(auto_ptr<A>*);

What's the significance of a pointer-to-auto_ptr?  I'd
understand what you meant if you wrote:

     void f(auto_ptr<A>);

instead.  I'm going to assume that's what you meant.

> It is very useful when wrapping interfaces which expects the
> user to create objects and give up ownership.

Sure, great.  It's a function-call-oriented thing.  Before
you object, read on.

>> Hmm, this is really very specific to calls, because it
>> *does* manage arguments and return values.  I really think
>> CallPolicy is better.  In any case I think we should
>> converge on this, one way or another; there will be more
>> languages, and you do want to be able to steal my users,
>> right? <wink>.  That'll be a lot easier if they see
>> familiar terminology ;-)
>
> I don't think it's specific to calls, but to all conversion
> of types between the languages. We can use policies when
> fetching values from lua, or when calling lua functions from
> c++:
>
> A* stolen_obj = object_cast<A*>(get_globals(L)["obj"],
> adopt(result));

What is result?  A placeholder?

could be spelled:

    std::auto_ptr<A> stolen
        = extract<std::auto_ptr<A> >(get_globals(L)["obj"]);

in Boost.Python.

[I assume this means that all of your C++ objects are held
within their lua wrappers by pointer.  I went to
considerable lengths to allow them to be held by-value,
though I'm not sure the efficiency gain is worth the cost
in flexibility.]

> call_function<void>(L, "f", stolen_obj) [ adopt(_1) ];

That's spelled:

  call_function<void>(L, "f", std::auto_ptr<A>(stolen));

I can begin to see the syntactic convenience of your way,
but I worry about the parameterizability.  In the first case
"result" is the only possible appropriate arg and in the 2nd
case it's "_1".

> And yeah, of course stealing your users is our goal. :)

I certainly hope so!  Likewise, I'm sure!

>> >> I'm not committed to the idea of a single registry.
>> >> In fact we've been discussing a hierarchical registry
>> >> system where converters are searched starting with a
>> >> local module registry and proceeding upward to the
>> >> package level and finally ending with a global
>> >> registry as a fallback.
>> >
>> > Right, that seems reasonable.
>>
>> Cool.  And let me also point out that if the module
>> doesn't have to collaborate with other modules, you don't
>> even need a registry lookup at static initialization
>> time.  A static data member of a class template is enough
>> to create an area of storage associated with a C++ type.
>> There's no central registry at all in that case.  I have
>> grave doubts about whether it's worth special-casing the
>> code for this, but it might make threading easier to cope
>> with.
>
> I can't see why it would be worth it. 

I like your attitude.

> If the module doesn't interact with other modules
> threading wouldn't be an issue?  So how could it make it
> easier?

I guess only in the case that the modules are loadedq
multiple times by different threads *and* the compiler
provides threadsafe static initializers, you wouldn't have
to worry about the central registry being modified while
someone was reading it.  A minor issue, really.  Mutexes
handle everything.

>> >> My big problem was trying to figure out a scheme for
>> >> assigning match quality.  C++ uses a kind of
>> >> "substitutaiblity" rule for resolving partial ordering
>> >> which seemed like a good way to handle things.  How do
>> >> you do it?
>> >
>> > We just let every converter return a value indicating
>> > how good the match was, where 0 is perfect match and -1
>> > is no match. When performing implicit conversions,
>> > every step in the conversions inreases the match value.
>> >
>> > Maybe I'm naive, but is there need for anything more
>> > complicated?
>>
>> Well, it's the "multimethod problem": consider base and
>> derived class formal arguments which both match an actual
>> argument, or int <--> float conversions.  How much do you
>> increase the match value by for a particular match?
>
> I don't know if I get this.

Are you familiar with the problems of multimethod
dispatching?  Google can help.

> We just increase the match value by one for every casting
> step that is needed for converting the types.

That seems to work for all the trivial cases, but the
problem is always phrased in more-complicated terms, I
presume for a reason.  See http://tinyurl.com/f5t6

One example of a place where it might not work is:

struct B {}; struct D : B {};

void f(B*, python::list)
void f(D*, std::vector<int>)

   >>> f(D(), [1, 2, 3])

I want this to choose the first overload, since it requires
only lvalue conversions.

Other links of interest:
  http://std.dkuug.dk/jtc1/sc22/wg21/docs/papers/2003/n1463.html
  http://tinyurl.com/f5vi

None of these uses such a trivial algorithm for rating
matches.  Coincidence?

> Right. Also, you can't get a pointer to all objects in lua,
> only "userdata" objects. If the object being converted is of
> primitive type, you can only access it directly from the
> stack with lua_toXXX() calls.

OK

>> > It doesn't seem that interesting to register different
>> > conversions for different states anymore. (at least not
>> > to me, but I could be wrong..). But if we where to
>> > increase the isolation of the registries, each state
>> > could just as well get their own registry.
>>
>> Let's continue poking at the issues until we get clarity.
>
> Yeah, I'll have to think about this for a bit, the whole
> registry thing is quite new to me.

OK.  FYI I'm going on vacation 6/26-7/6.  I find this
conversation really interesting, though, so I'll try to
keep an eye on it.

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com