[Numpy-discussion] Getting C-function pointers from Python to C

[Travis Oliphant]

>>>> Thoughts?
>>> 
>>> I really hope we can find some project-neutral common ground, so that lots of tools (Cython, f2py, numba, C extensions in NumPy and SciPy) can agree on how to "unbox callables".
>>> 
>>> A new extension type in NumPy would not fit this bill I feel. I've created a specification for this; if a number of projects (the ones mentioned above) agree on this or something similar and implement support, we could propose a PEP and do it properly once it has proven itself.
>>> 
>>> http://wiki.cython.org/enhancements/cep1000
>>> 
>>> In Cython, this may take the form
>>> 
>>> def call_callback(object func):
>>>    cdef double (*typed_func)(int)
>>>    typed_func = func
>>>    return typed_func(4)
>>> 
>>> ...it would be awesome if passing a Numba-compiled function just worked in this example.
>> 
>> Yes, I think we should go the Python PEP route.   However, it will take some time to see that to completion (especially with ctypes already in existence).   Dag, this would be a very good thing for you to champion however ;-)
> 
> I was NOT proposing a PEP.
> 
> The spec is created so that it can be implemented *now*, by the tools 
> "we" control (and still be very efficient). A "sci-PEP", if you will; a 
> mutual understanding between Cython, NumPy, numba (and ideally f2py, 
> which already has something similar, if anyone bothers).
> 
> When this is implemented in all the tools we care about, we can propose 
> something even nicer as a PEP, but that's far down the road; it'll be 
> another couple of years before I'm on Python 3.

Perfect :-)  We are on the same page....

> 
>> 
>> In the mean-time, I think we could do as Robert essentially suggested and just use Capsule Objects around an agreed-upon simple C-structure:
>> 
>> 	int   id   /* Some number that can be used as a "type-check" */
>> 	void *func;
>> 	char *string;
>> 
>> We can then just create some nice functions to go to and from this form in NumPy ctypeslib and then use this while the Python PEP gets written and adopted.
> 
> What is not clear to me is how one get from the Python callable to the 
> capsule.

This varies substantially based on the tool.   Numba would do it's work and create the capsule object using it's approach.   Cython would use a different approach.   

I would also propose to have in NumPy some basic functions that go back-and forth between this representation, ctypes, and any other useful representations that might emerge.  

> 
> Or do you simply intend to pass a non-callable capsule as an argument in 
> place of the callback?

I had simply intended to allow a non-callable capsule argument to be passed in instead of another call-back to any SciPy or NumPy function that can take a raw C-function pointer.

Thanks,

-Travis