[Cython] [cython-users] Cython .pxd introspection: listing defined constants

[Robert Bradshaw]

On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King <wking at drexel.edu> wrote:
> On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote:
>> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King <wking at drexel.edu> wrote:
>> > On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote:
>> >> On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King <wking at drexel.edu> wrote:
>> >> > However, the filename <-> module mapping is troublesome for backing
>> >> > externally-implemented Python modules (e.g. numpy).  If you wanted to
>> >> > write a .pxd file backing numpy.random, how would you go about getting
>> >> > your module installed in Cython/Includes/numpy/random.pxd or another
>> >> > path that cython would successfully match with `cimport numpy.random`?
>> >>
>> >> Note that extern blocks (by definition) declare where things come from.
>> >
>> > They declare where the .pxd file looks for .h files, but not where
>> > .pyx files look for the .pxd file.
>>
>> Sorry, I should have said extern blocks that make cdef class
>> declarations (such as our numpy.pxd).
>
> It doesn't look like there are cdef class declarations in numpy.pxd:
>
>    cython $ grep class Cython/Includes/numpy.pxd
>        ctypedef class numpy.dtype [object PyArray_Descr]:
>        ctypedef extern class numpy.flatiter [object PyArrayIterObject]:
>        ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]:
>        ctypedef class numpy.ndarray [object PyArrayObject]:
>        ctypedef extern class numpy.ufunc [object PyUFuncObject]:
>
> This still doesn't explain how .pxd files specify which external
> implemented Python modules they correspond to.

"numpy.dtype" is the fully qualified name of the class it's
declaring--the module is "numpy."

>> >> >> >      cdef public struct X:
>> >> >> >          int x
>> >> >> >          readonly int z
>> >> >> >          private int z
>> >> >> >
>> >> >> > I would perhaps say that non-Pythonable non-private members in public
>> >> >> > structs would be a compile error.
>> >> >>
>> >> >> +1, keep it safe at the beginning.
>> >> >
>> >> > -1, keep the code clean and the interface consistent ;).  I think the
>> >> > struct syntax should be identical to the class syntax, with the
>> >> > exception that you can't bind methods to structs.  That's the only
>> >> > real difference between structs and classes, isn't it?
>> >>
>> >> In C++, the only difference between structs and classes is that struct
>> >> members are public by default. (Not saying that C++ is always the
>> >> model to follow, but it gives precedent). And structs can have
>> >> function members, that's how to do OOP in C.
>> >
>> > Oh.  Even more reason to have identical struct and class handling in
>> > Cython ;).
>> >
>> > It is unclear to me what `cdef public struct` means.  I think it
>> > should mean "Python bindings can alter this struct's definition",
>> > which doesn't make sense.
>>
>> I think it should mean "this struct is accessible from Python (as X)"
>
> Wouldn't that be "cdef readonly struct X"?
>
>> > Shouldn't the syntax for public members be
>> >
>> >    cdef struct X:
>> >        cdef public:
>> >            int x
>> >            readonly int y
>> >            private int z
>>
>> -1 on nesting things like this. Rather than make a struct visible from
>> Python iff any of its members are,
>
> A struct is visible from python iff it is declared public or readonly:
>
>    cdef public struct X:
>        ...
>
> or
>
>    cdef readonly struct X:
>        ...
>
> I don't think the visibility of the struct as a whole should have any
> effect over the visibility of its members, so you should be able to
> specify member visibility explicitly with per-member granularity (as
> you currently can for classes).
>
> I was assuming that structs would be public by default (like classes),
> but that is obviously configurable.
>
>> I think it makes more sense to put
>> the declaration on the struct itself. We could support
>>
>> cdef public struct X:
>>     int x # public
>>
>> cdef readonly struct Y:
>>     int y # readonly
>>
>> cdef [private] struct Z:
>>     int z # private, as we don't even have Z in the Python namespace,
>> and no wrapper is created.
>
> The problems with this are:
>
> * It's differnent from how we handle the almost identical class case.

True it's different, but there are significant differences between
classes and structs in Cython, and this is the way things are now.

> * It makes it impossible to define, for example a public struct with
>  C-only attributes:
>
>    cdef public struct X:
>        cdef public int a
>        cdef private void* ptr

?

The above would work just fine. I was proposing that it would be
semantically equivalent to

    cdef public struct X:
        cdef int a
        cdef private void* ptr

Perhaps that was not clear. Consider the currently valid

cdef struct X:
    int a
    void* ptr

We can't make everything public by default, as this would break valid
code. (I don't think it's a good idea to make the default visibility
of a member be a function of its type if we can help it.) The original
proposal was to allow

cpdef struct X:
    ...

but making all fields private by default would be a bit useless, which
was why I was suggesting they be public. We could allow the the
modifier "cpdef readonly struct X" which would be the (probably less
common) case where all members were by default readonly, though
exposed to Python, rather than public. Within an exposed struct, of
course, any member could be declared as private/readonly/public
individually.

BTW, the "public" keyword is the wrong thing to use here, as that
actually controls name mangling and (c-level) symbol exporting. The
fact that means a different thing for members than for top-level
symbols isn't ideal, but at least it's unambiguous as members need not
be mangled.

> Obviously, public attributes of private structs should raise
> compile-time Cython errors.
>
>> >> > If safety with a new feature is a concern, a warning like
>> >> > "EXPERIMENTAL FEATURE" in the associated docs and compiler output
>> >> > should be sufficient.
>> >>
>> >> I think the point of "safe" is to start out with a compiler error, and
>> >> we can change our minds later, which is better than trying to make
>> >> legal statements illegal in the future.
>> >
>> > Ok, but I still don't understand why the cdefs were removed from the
>> > proposed structure members, when they are required for class
>> > definitions.
>>
>> Because structs can only have c members, so the cdef was entirely
>> redundant. They weren't really removed per say, it's just that with
>> the exception of cdef classes, "cdef ..." meant "a c declaration
>> follows."
>
> But cdef classes can also only have cdef members.
>
> I think it's better to keep cdef meaning "backed by C data", not
> necessarily "written using C syntax", since you're trying to do more
> with Cython, so it doesn't make sense to force C syntax.

It means both. Were I to start over, I would make "cdef int* a, b, c"
declare three pointers, but we're stuck with the C syntax we have. In
a struct "cdef" on members is entirely redundant (though I am not
strongly opposed to allowing it).

>> >> > That would be nice, since the C compiler would (I think) raise an error
>> >> > when you try to use an invalid <type> for macro value.
>> >>
>> >> Const is different than readonly, as readonly specifies the
>> >> python-level accessibility.
>> >
>> > Ah.  Sorry for all the c(p)def/qualifier confusion, but I'm trying to
>> > consolidate the way these are handled in Parsing/Nodes/Symtab and I
>> > want to make sure I don't implement the wrong interpretation.  Can you
>> > clarify how one knows if "public" means "expose a read/write Python
>> > interface to this object" or "expose this symbol to external C code"?
>>
>> Public has had several different meanings. I wish there were a spec
>> and full grammer for Cython, but there's not (yet?). The meaning is
>> implicit in the code, and there's enough users out there that we
>> should stay backwards compatible. It may be worth doing some
>> backwards-incompatible normalization before we hit 1.0, but compiling
>> the entire Python grammar is higher priority than that.
>
> Since I'm going to have lots of similar stuff (classes, enums,
> structs, unions) all with the same (hopefully) cdef/cpdef/visibility
> stuff for members, I'd like to consolidate now.  I will of course, add
> special-case code as necessary to support the current syntax, which
> can then be removed whenever you think it is appropriate, but writing
> separate, near-identical handlers for each type seems like a recipe
> for disaster ;).

Yes. For structs/unions, it's already almost always a StructOrUnion
object anyways. Classes are somewhat special.

> I'll look to the code for guidance on public, and try to work out the
> appropriate meaning during the parse phase.

Sounds good.

- Robert