boost::python, returning new PyObject references

[David Abrahams]

----- Original Message -----
From: "Arnaldur Gylfason" <arnaldur at decode.is>

> The reason I was thinking about overriding and virtual functions was
> twofold:
>
> a) Let's say I define a function that takes an (const) object & o.
>      If I would then call  o[i] (polymorphic)
>      If o is just an object , an exception is thrown
>      If o is a sequence, the generic sequence call is used
>      If o is a list or other concrete sequence object, the implementation
> in that concrete object is used.
>      (The implementation that can utilize the concrete object could be
> faster for instance).

So a) amounts to "it might be faster", right?

> I don't consider this very important but I feel it is okay to discuss
> things like this.

Of course.

> b)
> I have been toying with the idea of being able to use a list object
> (similar goes for dictionary) mylist thus:
>      mylist[i][j] = 20
>     instead of
>      list(mylist[i])[j] = 20
>
>    If  operator[] were virtual , and defined in object , and op[]  could
> return a (const) object &, this would be possible

Firstly, you can't return a reference (meaning T&) from operator[] (or from
any of the iterators, for that matter -- they are restricted to
InputIterator :-( ), because the Python objects don't store Ts... unless of
course T is PyObject*, but that misses the point of what we're trying to do
by managing Python objects with C++ wrappers.

Maybe you really want:

template <class T = ref> class list;

and then: list<> mylist;
          list<object> mylist;
or even
          list<sequence> mylist;

Or, maybe the right default for T in list<T> is object instead of ref. I
think that might be best. What do you think?

>      by :
>           if sequence::accepts(m_p)
>                return sequence ...
>           elif mapping::accepts(m_p)
>                return mapping ...
>           else
>                return object...
>
>  I can not see how it is possible to return a object & in a reliable way.

Right.

> Also, operator[](int) for sequence/list would clash with operator[](const
T
> & key) (w. T = int) for mapping/dictionary.

Not neccessarily. I started to go in the direction of using templated
arguments and relying on the existence of a to_python converter to make a
Python object out of it. For example, take a look at dictionary::operator[]:

    template <class Key>
    proxy operator[](const Key& key)
        { return this->operator[](make_ref(key)); }

That means you can index a dictionary with any C++ object for which there is
a to_python conversion.

> And, the accepts checks would be expensive (in each op[] call).

Are you sure? Most of these amount to only a few machine instructions. I
don't think most parts of Python are /that/ heavily optimized that you would
notice anything.

> Therefore, it is probably safe to forget this, but I would find it
> interesting if you have ideas about how to accomplish this.
>
>
> Now to your objects.hpp
>
> I have been looking at it and I think it looks pretty good.
> I am definately in favor of using ref as a smart pointer the way you do.
> One thing though, why not provide an explicit object(PyObject *)
> constructor?
> Some copy constructor calls would go away then.

Mostly because the interface to the Python 'C' API is inconsistent in how it
hands you objects. Sometimes you get a borrowed reference; sometimes Python
increments the reference count for you. Having a single place (ref::ref())
where ownership is adopted makes it less likely that you'll casually get it
wrong. Also, ref::ref() has explicit options for dealing with these borrowed
and zero references. If we had to put that in every object description
class, it would be costly. Probably the right thing to do is to define
wrapper functions for most of the Python 'C' API which only traffic in
managed references, so that this is less of an issue for users.

> Do you think sequence, list etc. should inherit from the basic object you
> use or the generic object?
> If it would inherit from the generic object I think the methods that
return
> ref should maybe return a generic object. That would make the following
> possible:
>      mylist[i].call_method(...)
>      my_list[i].set_attr(...)
>      ...
> instead of
>      object(mylist[i]).set_attr(...)
>      ...

I think we have come to the same conclusion ;-)

> Maybe object(mylist[i]).set_attr(...) is okay though.

No, I think you have the right idea.

> Then we always work with refs, and object,sequence,list,... are just
> interface wrappers that are wrapped around refs as needed.
> (I gathered that was the main idea behind your design).

Yes, that was it. However, it's important to provide a convenient interface,
especially if it's much safer than the 'C' interface. If it's not
convenient, people won't use it.

> After deciding if to use the generic object as the base class or the basic
> one, I'll rewrite what I wrote before in this design.

Great!

> About the name of the basic object (and naming in general):
>      You already have a base_object, which is the object itself not an
> interface wrapper. Maybe the terminology should be different to
>      differentiate between the two? (i for interface or something? (iList
,
> i_list , list_i, ... )

Naah, don't worry about base_object. It's in the detail subnamespace anyway
and doesn't have anything to do with these objects.

> About base_object.hpp:
> You use python_type as the name of a template parameter
>      template<class python_type>
>      struct base_object
> then you use the same name as a typedef:
>      typedef base_object<PyTypeObject> python_type

No I don't! the template parameter is PythonType. What source are you
reading?

> I would change the name of the template parameter (python_t, Tpython, ...)

Why?

=============================================================
If you get this message, it's because you're on my list of people who have
expressed an interest in Boost.Python development. If you'd like to be
removed from the list, either because of lack of interest or because of
email volume just tell me.
=============================================================