[Cython] [Python-Dev] C-level duck typing

[mark florisson]

On 16 May 2012 21:16, Dag Sverre Seljebotn <d.s.seljebotn at astro.uio.no> wrote:
> On Wed, 16 May 2012 20:49:18 +0100, mark florisson
> <markflorisson88 at gmail.com> wrote:
>>
>> On 16 May 2012 20:15, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>
>>> "Martin v. Löwis", 16.05.2012 20:33:
>>>>>
>>>>> Does this use case make sense to everyone?
>>>>>
>>>>> The reason why we are discussing this on python-dev is that we are
>>>>> looking
>>>>> for a general way to expose these C level signatures within the Python
>>>>> ecosystem. And Dag's idea was to expose them as part of the type
>>>>> object,
>>>>> basically as an addition to the current Python level tp_call() slot.
>>>>
>>>>
>>>> The use case makes sense, yet there is also a long-standing solution
>>>> already to expose APIs and function pointers: the capsule objects.
>>>>
>>>> If you want to avoid dictionary lookups on the server side, implement
>>>> tp_getattro, comparing addresses of interned strings.
>>>
>>>
>>> I think Martin has a point there. Why not just use a custom attribute on
>>> callables that hold a PyCapsule? Whenever we see inside of a Cython
>>> implemented function that an object variable that was retrieved from the
>>> outside, either as a function argument or as the result of a function
>>> call,
>>> is being called, we try to unpack a C function pointer from it on all
>>> assignments to the variable. If that works, we can scan for a suitable
>>> signature (either right away or lazily on first access) and cache that.
>>> On
>>> each subsequent call through that variable, the cached C function will be
>>> used.
>>>
>>> That means we'd replace Python variables that are being called by
>>> multiple
>>> local variables, one that holds the object and one for each C function
>>> with
>>> a different signature that it is being called with. We set the C function
>>> variables to NULL when the Python function variable is being assigned to.
>>> When the C function variable is NULL on call, we scan for a matching
>>> signature and assign it to the variable.  When no matching signature can
>>> be
>>> found, we set it to (void*)-1.
>>>
>>> Additionally, we allow explicit user casts of Python objects to C
>>> function
>>> types, which would then try to unpack the C function, raising a TypeError
>>> on mismatch.
>>>
>>> Assignments to callable variables can be expected to occur much less
>>> frequently than calls to them, so this will give us a good trade-off in
>>> most cases. I don't see why this kind of caching would be any slower
>>> inside
>>> of loops than what we were discussing so far.
>>>
>>> Stefan
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>> This works really well for local variables, but for globals, def
>> methods or callbacks as attributes, this won't work so well, as they
>> may be rebound at any time outside of the module scope. I think in
>
>
> +1. The python-dev discussion is pretty focused on the world of a manually
> written C extension. But code generation is an entirely different matter.
> Python puts in place pretty efficient boundaries against full-program static
> analysis, so there's really not much we can do.
>
> Here's some of my actual code I have for wrapping a C++ library:
>
> cdef class CallbackEventReceiver(BasicEventReceiver):
>    cdef object callback
>
>    def __init__(self, callback):
>        self.callback = callback
>
>    cdef dispatch_event(self, ...):
>        self.callback(...)
>
> The idea is that you can subclass BasicEventReceiver in Cython for speed,
> but if you want to use a Python callable then this converter is used.
>
> This code is very performance critical. And, the *loop* in question sits
> deep inside a C++ library.
>
> Good luck pre-acquiring the function pointer of self.callback in any useful
> way. Even if it is not exported by the class, that could be overridden by a
> subclass. I stress the fact that this is real world code by yours truly
> (unfortunately not open source, it wraps a closed source library).
>
> Yes, you can tell users to be mindful of this and make as much as possible
> local variables, introduce final modifiers and __nomonkey__ and whatnot, but
> that's a large price to pay to avoid hacking tp_flags.
>
> Dag
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Definitely. I personally prefer the metaclass approach, but it's an
irrelevant detail. If we go the tp_flags route, would we copy all the
interface information from the superclass into the subclass directly?
I think in any case we need a wrapper around PyType_Ready to inherit
the tp_flags bit (which would be automatic with a metaclass).