[Cython] [cython-users] Cython .pxd introspection: listing defined constants

Robert Bradshaw robertwb at math.washington.edu
Fri Feb 18 00:54:48 CET 2011

Forgot reply-all... didin't we have this discussion before about
making that the default for this list as it is by-far the most common
desired behavior?

On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King <wking at drexel.edu> wrote:
>> On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote:
>>> On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King <wking at drexel.edu> wrote:
>>> > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote:
>>> >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King <wking at drexel.edu> wrote:
>>> >> > What I'm missing is a way to bind the ModuleScope namespace to a name
>>> >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib,
>>> >> > name)` will work in expose.pyx.
>>> >>
>>> >> You have also hit into the thorny issue that .pxd files are used for
>>> >> many things. They may be pure C library declarations with no Python
>>> >> module backing, they may be declarations of (externally implemented)
>>> >> Python modules (such as numpy.pxd), or they may be declarations for
>>> >> Cython-implemented modules.
>>> >>
>>> >> Here's another idea, what if extern blocks could contain cpdef
>>> >> declarations, which would automatically generate a Python-level
>>> >> wrappers for the declared members (if possible, otherwise an error)?
>>> >
>>> > Ah, this sounds good!  Of the three .pxd roles you list above,
>>> > external Python modules (e.g. numpy) and Cython-implemented modules
>>> > (e.g. matched .pxd/.pyx) both already have a presence in Python-space.
>>> > What's missing is a way to give (where possible) declarations of
>>> > external C libraries a Python presence.  cpdef fills this hole nicely,
>>> > since its whole purpose is to expose Python interfaces to
>>> > C-based elements.
>>> In the case of external Python modules, I'm not so sure we want to
>>> monkey-patch our stuff in
>> I don't think any of the changes we are suggesting would require
>> changes to existing code, so .pxd-s with external implementations
>> wouldn't be affected unless they brough the changes upon themselves.
> Say, in numpy.pxd, I have
> cdef extern from "...":
>    cpdef struct obscure_internal_struct:
>        ...
> Do we add an "obscure_internal_struct" onto the (global) numpy module?
> What if it conflicts with a (runtime) name? This is the issue I'm
> bringing up.
>>> (and where would we do it--on the first import of a cimporting
>>> module?)
>> Compilation is an issue.  I think that .pxd files should be able to be
>> cythoned directly, since then they Cython can build any wrappers they
>> request.  If the file has a matching .pyx file, cythoning either one
>> should compile both together, since they'll produce a single Python
>> .so module.
> In this case, I think it may make more sense to consider cimporting
> stuff from .pyx files if no .pxd file is present. In any case, this is
> a big change. I don't like the idea of always creating such a module
> (it may be empty, or have name conflicts) nor the idea of
> conditionally compiling it (does one have to look at the contents and
> really understand Cython to see if a Python shadow will be created?)
>>> > A side effect of this cpdef change would be that now even bare .pxd
>>> > files (no matching .pyx) would have a Python presence,
>>> Where would it live? Would we just create this module (in essence,
>>> acting as if there was an empty .pyx file sitting there as well)? On
>>> this note, it may be worth pursuing the idea of a "cython helper"
>>> module where common code and objects could live.
>> I'm not sure exactly what you mean by "cython helper", but this sounds
>> like my 'bare .pyx can create a Python .so module idea above.
> I'm thinking of a place to put, e.g. the generator and bind-able
> function classes, which are now re-implemented in every module that
> uses them. I think there will be more cases like this in the future
> rather than less. C-level code could be #included and linked from
> "global" stores as well. However, that's somewhat tangential.
>>> > so You could do
>>> > something like
>>> >
>>> >   cimport mylib as mylib_c
>>> >   import mylib as mylib_py
>>> >   import sys
>>> >
>>> >   # Access through Python
>>> >   for name in dir(mylib_py):
>>> >       setattr(sys.modules[__name__], name, getattr(mylib_py, name))
>>> I think this smells worse than "import *"
>> Aha, thanks ;).  I was stuck in my old
>> .pxd-files-don't-create-modules-by-themselves mindset.  Obviously,
>> once they do, any Python code can access the contents directly and I
>> can throw out all this indirection.
>>> > However, from Parsing.py:2369:
>>> >
>>> >    error(pos, "C struct/union/enum cannot be declared cpdef")
>>> >
>>> > From pyrex_differences.rst:
>>> >
>>> >    If a function is declared :keyword:`cpdef` it can be called from
>>> >    and overridden by both extension and normal python subclasses.
>>> >
>>> > I believe the reason that cpdef-ed enums and similar are currently
>>> > illegal is confusion between "can be called from Python" and "can be
>>> > overridden from Python".
>>> The reason that error statement is there is because it had no meaning,
>>> so an error was better than just ignoring it.
>> Why does it have no meaning?  I understand that it's not implemented
>> yet, but a cpdef-ed enum or struct seems just as valid an idea as a
>> cpdef-ed method.
> We're only deciding what the meaning is now. I agree it's a valid
> idea, its just that no one had even considered it yet. (I'll also
> concede that it's a much more natural idea for someone who came to the
> language once cpdef was already implemented.)
>>> > Unions don't really have a Python parallel,
>>> They can be a cdef class wrapping the union type.
>> But I would think coercion would be difficult.  Unions are usually (in
>> my limited experience) for "don't worry about the type, just make sure
>> it fits in X bytes".  How would union->Python conversion work?
> There would be a wrapping type, e.g.
> cdef class MyUnion:
>    cdef union_type value
> with a bunch of setters/getters for the values, just like there are
> for structs. (In fact the same code would handle structs and unions).
> This is getting into the wrapper-generator territory, but I'm starting
> to think for simple things that might be worth it.
>>> >    cpdef struct Foo:
>>> >        cpdef public int intA
>>> >        cpdef readonly int intB
>>> >        cdef void *ptr
>>> >
>>> > We would both declare the important members of the C struct (as we can
>>> > already do in Cython) and also have Cython automatically generate a
>>> > Python class wrapping the struct (because of `cpdef struct`).  The
>>> > Python class would have:
>>> >
>>> > * Cython-generated getter/setter for intA (because of `cpdef public`)
>>> >  using the standard Python<->int coercion.
>>> > * Similar Cython-generated getter for int B (because of `cpdef
>>> >  readonly`).
>>> > * No Python access to ptr (standard C-access still possible through
>>> >  Cython).
>>> >
>>> > Doing something crazy like `cdef public void *ptr` would raise a
>>> > compile-time error.
>>> Yes, all of the above was exactly what I was proposing.
>>> > I'm definately willing to help out with this (if someone will point me
>>> > in the right direction),
>>> That would be great.
>> Ok, I think we're pretty much agreed ;).  I think that the next step
>> is to start working on implementations of:
>> * Stand alone .pxd -> Python module
> I'm not sure we're agreed on this one.
>> * Extending class cdef/cdpef/public/readonly handling to cover enums,
>>  stucts, and possibly unions.
> This seems like the best first step.
>> Problems with me getting started now:
>> * I don't know where I should start mucking about in the source.
> I would start by (1) creating/declaring cdef classes for cdef
> structs/unions/enums (not textually, within, e.g.
> CStructOrUnionDefNode.analyse_declarations(...) in Nodes.py) and then
> making sure they get added to the module's namespace (look at
> ModuleNode.py where the classes get added). The other files to look at
> would be Parsing.py, PyrexTypes.py and Symtab.py. Could either try to
> patch up parsing first, but I think it'd be more effective to make
> this the default during development, as then all the tests would
> exercise your code, and it's probably both easier to dive in there and
> would give you a better idea of how best to handle the parsing.
> Also, the creating of cpdef functions is a bit hackish (for historical
> reasons) and something we want to clean up, so I wouldn't go to great
> lengths to try and emulate that.
>> * I don't know how to handle things like dummy enums (perhaps by
>>  requiring all cdef-ed enums to be named).
> All enums in C are named.
>> * I'm going to go watch a movie ;).
> :)
>> It's good to be moving forward!
> Yep. You might want to consider creating a github fork.
> - Robert

More information about the cython-devel mailing list