[Python-3000] PyUnicodeObject implementation

[Stefan Behnel]

Martin v. Löwis wrote:
>>> Can you explain this a bit more? I presume you're talking about
>>> subclassing in C
>> Yes, I mentioned Cython above.
> 
> Can you please still elaborate? I have never used Cython before

Should have been clearer, sorry. C-level subtyping in Cython/Pyrex works as
follows. We create a new struct for the type that contains the parent-struct
as first field, and then we add the new attributes of the new type behind
that. This implies that this kind of subtyping is single inheritance (as
opposed to normal Python subclassing, which is the same in Pyrex/Cython and
Python). This currently works for all builtin types, except str. It results in
a very regular memory layout for extension types.

The way it's written in Pyrex/Cython is:

    cdef class MyListSubType(PyListObject):
        cdef int some_additional_int_field
        cdef my_struct* some_struct

        def __init__(self):
            self.some_struct = get_the_struct_pointer(...)
            self.some_additional_int_field = 1

PyListObject will become a struct member called "__pyx_base" in the new struct
for MyListSubType, and access to members of the base type does a straight

    self->__pyx_base-> (... possibly more __pyx_base derefs ...) -> field_name

The C compiler will make this a straight "self[index_of_field_name]" pointer
deref, unbeatable in speed.

The exact memory layout only needs to be available at C compile time. Also,
the exact members of the parent type(s) are not required at Cython compile
time (only those used in the code), as the C compiler will get them right when
it reads their header file.


> if it cannot efficiently subclass str, isn't that a bug in Cython?

I wouldn't mind letting Cython special case subtypes of str (or unicode in
Py3) *somehow*, as long as this "somewhow" proves to be a viable solution that
only applies to exactly those types *and* can be done realiably for subtypes
of subtypes. I'm just not aware of such a solution.


>>> I do note that the mechanisms that exist for supporting adding a __dict__
>>> to a str (in 2.x; or bytes in 3.x) or a tuple could be extended for other
>>> purposes.
>> I never looked into these, but this does not sound like it would impact
>> subclassing.
> 
> To me, the relationship is fairly straight: if you want to subclass a
> type, *all* you need is a way to place an __dict__ in the object, if
> it doesn't already have one. If the base object already has an __dict__,
> the layout of the subtype can be the same as the layout of the base
> type.

As long as you accept the dictionary indirection and type unpacking for
accessing fields even in the context of private C-level type members of an
extension type, which are currently accessible through straight pointers.

There is a huge performance difference between e.g. a) dereferencing a pointer
to a C int, and b) asking a dictionary for a name, have it find the result,
check if the result is empty, check if the result is a Python long or int (or
a pointer object, or whatever), unpack the result into a C int. Plus the need
to raise an exception in the error case, plus the Python-level visibility of
internal C-level fields (such as arbitrary pointers), plus the inability to do
this without holding the GIL. Plus the casting all over the place when it's
not a C int but a struct pointer, for example.


> Now, what Guido (probably) refers to is the implementation strategy
> used for adding __dict__ could be generalized for adding additional
> slots as well: for a variable-sized object (str or tuple), the
> dictoffset is negative, indicating that you have to count from the
> end of the object, not from the start, to find the slot.

This does sound interesting, but I will have to look into the implications. As
I said, it has to be a viable solution without (noticeable) impact on other
types. I'm not sure how this would interact with subtypes of subtypes, and
what the memory layout would be in that case.

Stefan