[Cython] CEP1000: Native dispatch through callables

[Dag Sverre Seljebotn]

On 04/19/2012 01:20 PM, Nathaniel Smith wrote:
> On Thu, Apr 19, 2012 at 11:56 AM, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no>  wrote:
>> I thought of some drawbacks of getfuncptr:
>>
>>   - Important: Doesn't allow you to actually inspect the supported
>> signatures, which is needed (or at least convenient) if you want to use an
>> FFI library or do some JIT-ing. So an iteration mechanism is still needed in
>> addition, meaning the number of things for the object to implement grows a
>> bit large. Default implementations help -- OTOH there really wasn't a major
>> drawback with the table approach as long as JIT's can just replace it?
>
> But this is orthogonal to the table vs. getfuncptr discussion. We're
> assuming that the table might be extended at runtime, which means you
> can't use it to determine which signatures are supported. So we need
> some sort of extra interface for the caller and callee to negotiate a
> type anyway. (I'm intentionally agnostic about whether it makes more
> sense for the caller or the callee to be doing the iterating... in
> general type negotiation could be quite complicated, and I don't think
> we know enough to get that interface right yet.)

Hmm. Right. Let's define an explicit goal for the CEP then.

What I care about at is getting the spec right enough such that, e.g., 
NumPy and SciPy, and other (mostly manually written) C extensions with 
slow development pace, can be forward-compatible with whatever crazy 
things Cython or Numba does.

There's 4 cases:

  1) JIT calls JIT (ruled out straight away)

  2) JIT calls static: Say that Numba wants to optimize calls to np.sin 
etc. without special-casing; this seem to require reading a table of 
static signatures

  3) Static calls JIT: This is the case when scipy.integrate routines 
calls a Numba callback and Numba generates a specialization for the 
dtype they explicitly needs. This calls for getfuncptr (but perhaps in a 
form which we can't quite determine yet?).

  4) Static calls static: Either table or getfuncptr works.

My gut feeling is go for 2) and 4) in this round => table.

The fact that the table can be extended at runtime is then not really 
relevant -- perhaps there will be an API to trigger that in the future, 
but it can't really be made use of today.

> The other other option would be to go to the far other end of
> simplicity, and just forget for now about allowing multiple signatures
> in the same object. Do signature selection by having the user select
> one explicitly:
>
> @cython.inline
> def square(x):
>      return x * x
>
> # .specialize is an un-standardized Cython interface
> # square_double is an object implementing the standardized C-callable interface
> square_double = square.specialize("d->d")
> scipy.integrate.quad(square_double)
>
> That'd be enough to get started, and doesn't rule out later extensions
> that do automatic type selection, once we have more experience.

Well, I want np.sin to replace "cdef extern from 'math.h'", and then 
this seems to be needed... at least the possibility to have both "d->d" 
and "O->O".

Dag