[Cython] [cython-users] Cython .pxd introspection: listing defined constants

[Robert Bradshaw]

On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King <wking at drexel.edu> wrote:
> On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote:
>> On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King <wking at drexel.edu> wrote:
>>
>> >> > If you try to override anything in a .so compiled module at runtime,
>> >> > you'd get the same kind of error you currently do trying to rebind a
>> >> > compiled class' method.
>> >>
>> >> That's the desired behavior for statically-bound globals, but
>> >> implementing it is not so trivial.
>> >
>> > It's currently implemented for classes.  Are modules that different
>> > from classes?
>> >
>> >    >>> import types
>> >    >>> types.ModuleType.__class__.__mro__
>> >                (<type 'type'>, <type 'object'>)
>> >
>> > So you've got your standard __setattr__ to override with an
>> > error-message generator.  What is the implementation difficulty?
>>
>> You can't just add a __setattr__ to a module and have it work, it's a
>> class-level slot. And you don't want to modify all module classes. To
>> do this you have to subclass module itself and insert that in place
>> during import.
>
> So annoying but possible?  Not particularly critical either way,
> though, since you could always say "don't rebind module-level stuff"
> in a project's docs, even if you don't enforce that at the Cython
> level.

It's still something I'd like to do. This could also be used to allow
fast access to module globals and easier sharing of declarations.

>> > I am also unclear about the distinction between cdef and cpdef
>> > (perhaps this should be a new thread?).
>> >
>> > cpdef means "I'm declaring something with C and Python interfaces.
>> >  The Python interface is a thin wrapper which can be rebound to an
>> >  object of any type, leaving the static C interface inaccessible from
>> >  Python."
>>
>> No, you can't re-bind cdef class methods. Despite Cython's attempt to
>> homogenize things for the user, extension classes are quite different
>> than "normal" Python classes. This is a Python C/API issue.
>
> Do you mean `cpdef class methods`?  If so, you're right:
>
>    $ cat square.pyx
>    cdef class A (object):
>        cdef public int value
>
>        cpdef square(self):
>            return self.value**2
>    $ python -c 'import pyximport; pyximport.install(); import square;
>                square.A.square = lambda self: self.value'
>    Traceback (most recent call last):
>      File "<string>", line 1, in <module>
>    TypeError: can't set attributes of built-in/extension type 'square.A'
>
> You can't override them in instances either:
>
>    $ python -c 'import pyximport; pyximport.install(); import square;
>    a = square.A(); a.square = lambda self: self.value'
>    Traceback (most recent call last):
>      File "<string>", line 1, in <module>
>    AttributeError: 'square.A' object attribute 'square' is read-only
>
> But you can override them in subclasses, which is, I suppose, the
> point of cpdef for methods.

Yes, otherwise they would have been pretty easy to implement :).

>> > cdef [private] means "I'm declaring something that only has a C
>> >  interface."
>> > cdef public means "I'm declaring something with C and Python
>> >  interfaces backed by C data.  Python code can alter the C data."
>> > cdef readonly means "I'm declaring something with C and Python
>> >  interfaces backed by C data.  Python code cannot alter the C data."
>>
>> cdef means "back this by a C variable"
>
> Ah, ok.  That makes more sense.
>
>> > However, the filename <-> module mapping is troublesome for backing
>> > externally-implemented Python modules (e.g. numpy).  If you wanted to
>> > write a .pxd file backing numpy.random, how would you go about getting
>> > your module installed in Cython/Includes/numpy/random.pxd or another
>> > path that cython would successfully match with `cimport numpy.random`?
>>
>> Note that extern blocks (by definition) declare where things come from.
>
> They declare where the .pxd file looks for .h files, but not where
> .pyx files look for the .pxd file.

Sorry, I should have said extern blocks that make cdef class
declarations (such as our numpy.pxd).

>> >> >      cdef public struct X:
>> >> >          int x
>> >> >          readonly int z
>> >> >          private int z
>> >> >
>> >> > I would perhaps say that non-Pythonable non-private members in public
>> >> > structs would be a compile error.
>> >>
>> >> +1, keep it safe at the beginning.
>> >
>> > -1, keep the code clean and the interface consistent ;).  I think the
>> > struct syntax should be identical to the class syntax, with the
>> > exception that you can't bind methods to structs.  That's the only
>> > real difference between structs and classes, isn't it?
>>
>> In C++, the only difference between structs and classes is that struct
>> members are public by default. (Not saying that C++ is always the
>> model to follow, but it gives precedent). And structs can have
>> function members, that's how to do OOP in C.
>
> Oh.  Even more reason to have identical struct and class handling in
> Cython ;).
>
> It is unclear to me what `cdef public struct` means.  I think it
> should mean "Python bindings can alter this struct's definition",
> which doesn't make sense.

I think it should mean "this struct is accessible from Python (as X)"

> Shouldn't the syntax for public members be
>
>    cdef struct X:
>        cdef public:
>            int x
>            readonly int y
>            private int z

-1 on nesting things like this. Rather than make a struct visible from
Python iff any of its members are, I think it makes more sense to put
the declaration on the struct itself. We could support

cdef public struct X:
    int x # public

cdef readonly struct Y:
    int y # readonly

cdef [private] struct Z:
    int z # private, as we don't even have Z in the Python namespace,
and no wrapper is created.

>> > If safety with a new feature is a concern, a warning like
>> > "EXPERIMENTAL FEATURE" in the associated docs and compiler output
>> > should be sufficient.
>>
>> I think the point of "safe" is to start out with a compiler error, and
>> we can change our minds later, which is better than trying to make
>> legal statements illegal in the future.
>
> Ok, but I still don't understand why the cdefs were removed from the
> proposed structure members, when they are required for class
> definitions.

Because structs can only have c members, so the cdef was entirely
redundant. They weren't really removed per say, it's just that with
the exception of cdef classes, "cdef ..." meant "a c declaration
follows."

>> > That would be nice, since the C compiler would (I think) raise an error
>> > when you try to use an invalid <type> for macro value.
>>
>> Const is different than readonly, as readonly specifies the
>> python-level accessibility.
>
> Ah.  Sorry for all the c(p)def/qualifier confusion, but I'm trying to
> consolidate the way these are handled in Parsing/Nodes/Symtab and I
> want to make sure I don't implement the wrong interpretation.  Can you
> clarify how one knows if "public" means "expose a read/write Python
> interface to this object" or "expose this symbol to external C code"?

Public has had several different meanings. I wish there were a spec
and full grammer for Cython, but there's not (yet?). The meaning is
implicit in the code, and there's enough users out there that we
should stay backwards compatible. It may be worth doing some
backwards-incompatible normalization before we hit 1.0, but compiling
the entire Python grammar is higher priority than that.

- Robert