[Cython] How to define C-consts in python module scope

[Robert Bradshaw]

On Wed, Jul 20, 2011 at 12:53 PM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 20 July 2011 21:44, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>> On 20 July 2011 16:27, mark florisson <markflorisson88 at gmail.com> wrote:
>>> On 20 July 2011 21:13, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>> On 20 July 2011 15:32, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>> On 20 July 2011 20:04, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>>> On 20 July 2011 13:51, mark florisson <markflorisson88 at gmail.com> wrote:
>>>>>>> On 20 July 2011 18:06, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>>>>> On 19 July 2011 20:48, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>>>>>>>> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>>>>>>> On 19 July 2011 02:24, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>>>>>>>>> 2011/7/18 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>> Trevor King and I discussed this quite a while back, but every time I
>>>>>>>>>>>> got around to looking at his code (I don't think he ever created a
>>>>>>>>>>>> formal pull request) something came up. The idea was that we could
>>>>>>>>>>>> support cpdef structs and extern functions as well.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> That's interesting, I think I shouldn't hurry with my pull request.
>>>>>>>>>>>
>>>>>>>>>>> 2011/7/19 Robert Bradshaw <robertwb at math.washington.edu>:
>>>>>>>>>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>>>>>>>>>>>>> My suggestion is
>>>>>>>>>>>>>
>>>>>>>>>>>>>  cdef exposed enum:
>>>>>>>>>>>>>    ...
>>>>>>>>>>>>
>>>>>>>>>>>> I agree, public is an overloaded word. This meaning is analogous to
>>>>>>>>>>>> its use for cdef class members. Perhaps we should use "api" for api
>>>>>>>>>>>> generation, and public used for Python-level access, with cpdef being
>>>>>>>>>>>> the preferred form for declaring Python-accessible types.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It seems to me that cpdef is more cythonic but exposed could be used
>>>>>>>>>>> in this case:
>>>>>>>>>>>
>>>>>>>>>>> cdef extern from "ev.h":
>>>>>>>>>>>    exposed enum:
>>>>>>>>>>>        EV_READ
>>>>>>>>>>>        EV_WRITE
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> And what about this?
>>>>>>>>>>
>>>>>>>>>> cdef extern from "ev.h":
>>>>>>>>>>   cpdef enum:
>>>>>>>>>>       EV_READ
>>>>>>>>>>       EV_WRITE
>>>>>>>>>
>>>>>>>>> Yep, exactly.
>>>>>>>>>
>>>>>>>>>> BTW, how is this supposed to work with *.pxd files? I think the values
>>>>>>>>>> will be exposed just in the matching implementation .pyx file, and not
>>>>>>>>>> with cimport, right?
>>>>>>>>>
>>>>>>>>> It would be an error unless there's an corresponding .pyx file (which
>>>>>>>>> could be empty). The idea is that one could also define extern cpdef
>>>>>>>>> functions and structs and wrappers would be provided.
>>>>>>>>>
>>>>>>>>
>>>>>>>> It would be an error? What do you mean? if you cpdef enum in foo.pxd,
>>>>>>>> and have foo.pyx, then the enumerations should be exposed in the 'foo'
>>>>>>>> module. But then if you "cimport foo" in bar.pyx, that should succeed
>>>>>>>> but the enumerations should not be exposed in the "bar" module... Am I
>>>>>>>> missing something?
>>>>>>>>
>>>>>>>
>>>>>>> I believe Robert confirmed what you said and added to that that it
>>>>>>> would be an error to have a cpdef extern enum in a .pxd without a
>>>>>>> corresponding .pyx file.
>>>>>>>
>>>>>>
>>>>>> But what an "error" means? Cython is going to fail compilation?
>>>>>> Suppose I develop a package, and at setup.py install time my package
>>>>>> installs somewhere a .pxd full of definitions in such a way that
>>>>>> third-party code can cimport it. Then, while writing the third party
>>>>>> code, you cimport the my package .pxd, and obviously there is no .pyx
>>>>>> for my package, as you are writing YOUR code. What's going to happen
>>>>>> in this case?
>>>>>
>>>>> If you want to use cpdef in your .pxd, it means you want to expose
>>>>> those to Python. This means they must be importable from Python, which
>>>>> means you need an extension module that exposes the constants, which
>>>>> means you need a .pyx. So you ship a (possibly empty) .pyx file along
>>>>> with your .pxd. If you don't want to expose them to Python but only to
>>>>> Cython, don't use cpdef. This compile time error would be issued
>>>>> whenever a user does a cimport of a .pxd file that has a cpdef enum
>>>>> with no corresponding .pyx file, i.e. the pxd is unusable, you
>>>>> wouldn't ship it in the first place.

Alternatively it would be a runtime error when trying to import the
objects. I suppose if cimport does not inject anything into the python
namespace, then "cimport" could just work and "import" would work iff
the module is available at runtime.

>>>> So supose I want to write a small wrapper for dlopen(), then I do:
>>>>
>>>> # dl.pxd
>>>> cdef from extern from "dlfcn.h":
>>>>    cpdef enum: RTLD_GLOBAL
>>>>    void * dlopen()(const char *filename, int flag);
>>>>
>>>> # dl.pyx
>>>> def open(filename, mode):
>>>>    cdef void *handle = c_dlopen(fiename,handle)
>>>>    return <long>handle
>>>>
>>>> Then the "dl" module namespace contains "dlopen" and "RTLD_GLOBAL"
>>>>
>>>> Now, suppose I code my setup.py to build the ext module and install
>>>> dl.pxd as package data.
>>>>
>>>> Now a user runs "pip install dl" and installs my package. She wants to
>>>> reuse my .pxd (installed somewhere) for calling dlopen() in its own
>>>> Cython code:
>>>>
>>>> # user.pyx
>>>> from dl cimport RTLD_GLOBAL, c_dlopen
>>>> dlopen("somelibrary.so", RTLD_GLOBAL)
>>>>
>>>>
>>>> So the user code is going to fail? Then I cannot take advantage of
>>>> cpdef enum for developing my  "dl" module, I have to go back to
>>>> manually exposing the enumerations
>>>>
>>>
>>> Why can't you ship both dl.pxd and dl.pyx as package data?
>>>
>>
>> This is like asking to provide a C source in order to being able to
>> #include "someheader.h" in other C sources. End users do not need
>> implementation source code, they need declarations,interface,header
>> files.

Because we all love link errors :).

> I personally don't really mind, but what about not allowing cpdef in
> .pxds for enums, but only in .pyx files:
>
> from mypxd cimport my_extern_enum
> cpdef enum my_extern_enum
>
> Of course, the usual
>
> cpdef enum my_enum:
>   ...
>
> would still be allowed in .pyx files.

Hmm... that's an idea, though it means declaring things twice. Also,
what about big lists of enums in, e.g. numpy.pxd or the system
headers?

I'm perhaps OK with them being like cdef classes--the .pyx file is not
needed at compile time it's a runtime error on import if the
corresponding module is not available.

- Robert