[Cython] CEP1000: Native dispatch through callables

mark florisson markflorisson88 at gmail.com
Sun Apr 15 13:30:27 CEST 2012

On 15 April 2012 07:26, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 14.04.2012 23:15:
>> On 14 April 2012 22:02, Stefan Behnel wrote:
>>> Dag Sverre Seljebotn, 14.04.2012 21:08:
>>>>  * TBD: Support for Cython-specific constructs like memoryview slices
>>>>    (so that arrays with strides and shape can be passed faster than
>>>>    passing an {{{"O"}}}).
>>> Is this really Cython specific or would a generic Py_buffer struct work?
>> That could work through simple unboxing wrapper functions, but it
>> would add some overhead, specifically because it would have to check
>> the buffer's object, and if it didn't exist or was not a memoryview
>> object, it would have to create one (checking whether something is a
>> memoryview object would also be a pain, as each module has a different
>> memoryview type). That could still be feasible for interaction with
>> Cython functions from non-Cython code.
> Hmm, I don't get it. Isn't the overhead always there when a memory view is
> requested in the signature? You'd have to create one for each call and that
> seriously hurts the efficiency. Is that a common use case? Why would you
> want to do more than passing unboxed buffers?
> Stefan
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

So, if you're going to accept Py_buffer *buf (which is useful in
itself), then to use memoryviews you have to copy over some
shape/strides/suboffsets and the data pointer, which it not a big
deal. But you also want a memoryview object associated with the
memoryview slice, that keeps things around like the format string,
function pointers to convert the dtype to and from Python objects and
a reference (acquisition) count or a lock in case atomics are not
supported by the compiler (or Cython doesn't know about the compiler).
So if buf->obj is not a memoryview object, it will have to create one
in the callee, and the caller will have to convert a slice to a new
Py_buffer struct.

Arguably, the memoryview implementation is not optimal, it should have
a memoryview struct with that data, making it somewhat less expensive.

Finally, what are the semantics for Py_buffer? Will the callee own the
buffer, or will it borrow it? If they will borrow, then the compiler
will have to figure out whether it will need to own it (or be slower
and always own it), and acquire the buffer through buf->obj. At least
it won't have to validate the buffer, which is the most expensive
I think in many cases you want to borrow though, but if you want to
always own, the caller could do something more efficient if
releasebuffer is not implemented, like simply incref buf->obj and pass
in a pointer to a copy of the Py_buffer. I think borrowing is probably
the easiest and most sane way though.

More information about the cython-devel mailing list