[Cython] [cython-users] Cython .pxd introspection: listing defined constants

[W. Trevor King]

On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote:
> On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King <wking at drexel.edu> wrote:
> > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote:
> >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King <wking at drexel.edu> wrote:
> >> > What I'm missing is a way to bind the ModuleScope namespace to a name
> >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib,
> >> > name)` will work in expose.pyx.
> >>
> >> You have also hit into the thorny issue that .pxd files are used for
> >> many things. They may be pure C library declarations with no Python
> >> module backing, they may be declarations of (externally implemented)
> >> Python modules (such as numpy.pxd), or they may be declarations for
> >> Cython-implemented modules.
> >>
> >> Here's another idea, what if extern blocks could contain cpdef
> >> declarations, which would automatically generate a Python-level
> >> wrappers for the declared members (if possible, otherwise an error)?
> >
> > Ah, this sounds good!  Of the three .pxd roles you list above,
> > external Python modules (e.g. numpy) and Cython-implemented modules
> > (e.g. matched .pxd/.pyx) both already have a presence in Python-space.
> > What's missing is a way to give (where possible) declarations of
> > external C libraries a Python presence.  cpdef fills this hole nicely,
> > since its whole purpose is to expose Python interfaces to
> > C-based elements.
> 
> In the case of external Python modules, I'm not so sure we want to
> monkey-patch our stuff in

I don't think any of the changes we are suggesting would require
changes to existing code, so .pxd-s with external implementations
wouldn't be affected unless they brough the changes upon themselves.

> (and where would we do it--on the first import of a cimporting
> module?)

Compilation is an issue.  I think that .pxd files should be able to be
cythoned directly, since then they Cython can build any wrappers they
request.  If the file has a matching .pyx file, cythoning either one
should compile both together, since they'll produce a single Python
.so module.

> > A side effect of this cpdef change would be that now even bare .pxd
> > files (no matching .pyx) would have a Python presence,
> 
> Where would it live? Would we just create this module (in essence,
> acting as if there was an empty .pyx file sitting there as well)? On
> this note, it may be worth pursuing the idea of a "cython helper"
> module where common code and objects could live.

I'm not sure exactly what you mean by "cython helper", but this sounds
like my 'bare .pyx can create a Python .so module idea above.

> > so You could do
> > something like
> >
> >   cimport mylib as mylib_c
> >   import mylib as mylib_py
> >   import sys
> >
> >   # Access through Python
> >   for name in dir(mylib_py):
> >       setattr(sys.modules[__name__], name, getattr(mylib_py, name))
> 
> I think this smells worse than "import *"

Aha, thanks ;).  I was stuck in my old
.pxd-files-don't-create-modules-by-themselves mindset.  Obviously,
once they do, any Python code can access the contents directly and I
can throw out all this indirection.

> > However, from Parsing.py:2369:
> >
> >    error(pos, "C struct/union/enum cannot be declared cpdef")
> >
> > From pyrex_differences.rst:
> >
> >    If a function is declared :keyword:`cpdef` it can be called from
> >    and overridden by both extension and normal python subclasses.
> >
> > I believe the reason that cpdef-ed enums and similar are currently
> > illegal is confusion between "can be called from Python" and "can be
> > overridden from Python".
> 
> The reason that error statement is there is because it had no meaning,
> so an error was better than just ignoring it.

Why does it have no meaning?  I understand that it's not implemented
yet, but a cpdef-ed enum or struct seems just as valid an idea as a
cpdef-ed method.

> > Unions don't really have a Python parallel,
> 
> They can be a cdef class wrapping the union type.

But I would think coercion would be difficult.  Unions are usually (in
my limited experience) for "don't worry about the type, just make sure
it fits in X bytes".  How would union->Python conversion work?

> >    cpdef struct Foo:
> >        cpdef public int intA
> >        cpdef readonly int intB
> >        cdef void *ptr
> >
> > We would both declare the important members of the C struct (as we can
> > already do in Cython) and also have Cython automatically generate a
> > Python class wrapping the struct (because of `cpdef struct`).  The
> > Python class would have:
> >
> > * Cython-generated getter/setter for intA (because of `cpdef public`)
> >  using the standard Python<->int coercion.
> > * Similar Cython-generated getter for int B (because of `cpdef
> >  readonly`).
> > * No Python access to ptr (standard C-access still possible through
> >  Cython).
> >
> > Doing something crazy like `cdef public void *ptr` would raise a
> > compile-time error.
> 
> Yes, all of the above was exactly what I was proposing.
> 
> > I'm definately willing to help out with this (if someone will point me
> > in the right direction),
> 
> That would be great.

Ok, I think we're pretty much agreed ;).  I think that the next step
is to start working on implementations of:

* Stand alone .pxd -> Python module
* Extending class cdef/cdpef/public/readonly handling to cover enums,
  stucts, and possibly unions.

Problems with me getting started now:

* I don't know where I should start mucking about in the source.
* I don't know how to handle things like dummy enums (perhaps by
  requiring all cdef-ed enums to be named).
* I'm going to go watch a movie ;).

It's good to be moving forward!

-- 
This email may be signed or encrypted with GPG (http://www.gnupg.org).
The GPG signature (if present) will be attached as 'signature.asc'.
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20110217/8cea5a1c/attachment.pgp>