[Cython] [cython-users] Cython .pxd introspection: listing defined constants

[Robert Bradshaw]

On Sat, Feb 19, 2011 at 6:32 PM, W. Trevor King <wking at drexel.edu> wrote:
> On Sat, Feb 19, 2011 at 04:41:27PM -0800, Robert Bradshaw wrote:
>> On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King <wking at drexel.edu> wrote:
>> > On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote:
>> >> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King <wking at drexel.edu> wrote:
>> >> > On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote:
>> >> >> On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King <wking at drexel.edu> wrote:
>> >> >> > However, the filename <-> module mapping is troublesome for backing
>> >> >> > externally-implemented Python modules (e.g. numpy).  If you wanted to
>> >> >> > write a .pxd file backing numpy.random, how would you go about getting
>> >> >> > your module installed in Cython/Includes/numpy/random.pxd or another
>> >> >> > path that cython would successfully match with `cimport numpy.random`?
>> >> >>
>> >> >> Note that extern blocks (by definition) declare where things come from.
>> >> >
>> >> > They declare where the .pxd file looks for .h files, but not where
>> >> > .pyx files look for the .pxd file.
>> >>
>> >> Sorry, I should have said extern blocks that make cdef class
>> >> declarations (such as our numpy.pxd).
>> >
>> > It doesn't look like there are cdef class declarations in numpy.pxd:
>> >
>> >    cython $ grep class Cython/Includes/numpy.pxd
>> >        ctypedef class numpy.dtype [object PyArray_Descr]:
>> >        ctypedef extern class numpy.flatiter [object PyArrayIterObject]:
>> >        ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]:
>> >        ctypedef class numpy.ndarray [object PyArrayObject]:
>> >        ctypedef extern class numpy.ufunc [object PyUFuncObject]:
>> >
>> > This still doesn't explain how .pxd files specify which external
>> > implemented Python modules they correspond to.
>>
>> "numpy.dtype" is the fully qualified name of the class it's
>> declaring--the module is "numpy."
>
> Hmm, that means that
>
>    cimport numpy
>    a = numpy.numpy.ndarray
>
> also compiles.

As does

    a = numpy.foo.x.y.z

though perhaps that should be a compile-time error. To get the type,
you have to write

    cimport numpy
    a = numpy.ndarray.

> Wouldn't it make more sense to mirror numpy's module
> structure when backing it?  You do have nested .pxd modules for
> cpython, libc, etc.

Yes, in fact that's what we do, but that's not required (e.g. you can
have these extern declarations in .pyx files)

>> >> >> >> >      cdef public struct X:
>> >> >> >> >          int x
>> >> >> >> >          readonly int z
>> >> >> >> >          private int z
>> >> >> >> >
>> >> >> >> > I would perhaps say that non-Pythonable non-private members in public
>> >> >> >> > structs would be a compile error.
>> >> >> >>
>> >> >> >> +1, keep it safe at the beginning.
>> >> >> >
>> >> >> > -1, keep the code clean and the interface consistent ;).  I think the
>> >> >> > struct syntax should be identical to the class syntax, with the
>> >> >> > exception that you can't bind methods to structs.  That's the only
>> >> >> > real difference between structs and classes, isn't it?
>> >> >>
>> >> >> In C++, the only difference between structs and classes is that struct
>> >> >> members are public by default. (Not saying that C++ is always the
>> >> >> model to follow, but it gives precedent). And structs can have
>> >> >> function members, that's how to do OOP in C.
>> >> >
>> >> > Oh.  Even more reason to have identical struct and class handling in
>> >> > Cython ;).
>> >> >
>> >> > It is unclear to me what `cdef public struct` means.  I think it
>> >> > should mean "Python bindings can alter this struct's definition",
>> >> > which doesn't make sense.
>> >>
>> >> I think it should mean "this struct is accessible from Python (as X)"
>> >
>> > Wouldn't that be "cdef readonly struct X"?
>> >
>> >> > Shouldn't the syntax for public members be
>> >> >
>> >> >    cdef struct X:
>> >> >        cdef public:
>> >> >            int x
>> >> >            readonly int y
>> >> >            private int z
>> >>
>> >> -1 on nesting things like this. Rather than make a struct visible from
>> >> Python iff any of its members are,
>> >
>> > A struct is visible from python iff it is declared public or readonly:
>> >
>> >    cdef public struct X:
>> >        ...
>> >
>> > or
>> >
>> >    cdef readonly struct X:
>> >        ...
>> >
>> > I don't think the visibility of the struct as a whole should have any
>> > effect over the visibility of its members, so you should be able to
>> > specify member visibility explicitly with per-member granularity (as
>> > you currently can for classes).
>> >
>> > I was assuming that structs would be public by default (like classes),
>> > but that is obviously configurable.
>> >
>> >> I think it makes more sense to put
>> >> the declaration on the struct itself. We could support
>> >>
>> >> cdef public struct X:
>> >>     int x # public
>> >>
>> >> cdef readonly struct Y:
>> >>     int y # readonly
>> >>
>> >> cdef [private] struct Z:
>> >>     int z # private, as we don't even have Z in the Python namespace,
>> >> and no wrapper is created.
>> >
>> > The problems with this are:
>> >
>> > * It's differnent from how we handle the almost identical class case.
>>
>> True it's different, but there are significant differences between
>> classes and structs in Cython, and this is the way things are now.
>
> Ah, well.  I suppose I'll leave the Parser pretty much alone.  Classes
> and structs will end up being much closer on the Python interface
> side, perhaps the syntax will grow closer in future Cythons...

Well, we certainly can't get rid of the old syntax without a lot of
thought and justification.

>> > * It makes it impossible to define, for example a public struct with
>> >  C-only attributes:
>> >
>> >    cdef public struct X:
>> >        cdef public int a
>> >        cdef private void* ptr
>>
>> ?
>>
>> The above would work just fine. I was proposing that it would be
>> semantically equivalent to
>>
>>     cdef public struct X:
>>         cdef int a
>>         cdef private void* ptr
>>
>> Perhaps that was not clear. Consider the currently valid
>>
>> cdef struct X:
>>     int a
>>     void* ptr
>>
>> We can't make everything public by default, as this would break valid
>> code. (I don't think it's a good idea to make the default visibility
>> of a member be a function of its type if we can help it.)
>
> Oh, I though (for some reason) that you were ruling out member-level
> visibility adjustments.  Using the struct visibility to set
> member-level defaults is fine.
>
>> BTW, the "public" keyword is the wrong thing to use here, as that
>> actually controls name mangling and (c-level) symbol exporting. The
>> fact that means a different thing for members than for top-level
>> symbols isn't ideal, but at least it's unambiguous as members need not
>> be mangled.
>
> "public" means "read/write Python interface" for struct attributes,

"public" means "C-visible" for global names, "read/write Python
interface" for (cdef class) members.

> I
> was assuming that would also apply to struct members acording to
> tutorial/cdef_classes.  Should I use a different visibility name in
> structs and unions?

No, I think public/readonliy/private is a perfectly fine thing for
struct members. It's just that declaring the struct itself as public
already has a different meaning.

>> >> >> > If safety with a new feature is a concern, a warning like
>> >> >> > "EXPERIMENTAL FEATURE" in the associated docs and compiler output
>> >> >> > should be sufficient.
>> >> >>
>> >> >> I think the point of "safe" is to start out with a compiler error, and
>> >> >> we can change our minds later, which is better than trying to make
>> >> >> legal statements illegal in the future.
>> >> >
>> >> > Ok, but I still don't understand why the cdefs were removed from the
>> >> > proposed structure members, when they are required for class
>> >> > definitions.
>> >>
>> >> Because structs can only have c members, so the cdef was entirely
>> >> redundant. They weren't really removed per say, it's just that with
>> >> the exception of cdef classes, "cdef ..." meant "a c declaration
>> >> follows."
>> >
>> > But cdef classes can also only have cdef members.
>> >
>> > I think it's better to keep cdef meaning "backed by C data", not
>> > necessarily "written using C syntax", since you're trying to do more
>> > with Cython, so it doesn't make sense to force C syntax.
>>
>> It means both. Were I to start over, I would make "cdef int* a, b, c"
>> declare three pointers, but we're stuck with the C syntax we have...
>>
>> In a struct "cdef" on members is entirely redundant
>
> As it is in a cdef class.  If you don't want it there for struct/union
> members, though, then I'll do it that way.

We're sticking with backwards compatibility.

- Robert