[Numpy-discussion] Releasing the GIL in ufuncs dealing with object arrays

[Joris Van den Bossche]

Hi Sebastian,

Thanks for the answer!

On Mon, 19 Aug 2019 at 17:57, Sebastian Berg <sebastian at sipsolutions.net>
wrote:

> ...
>
> Hmmm, interesting use case. No, I do not think there currently is a
> reasonable way to do this (I think there may be ways to hack it). Even
> when all access to the objects is safe by itself, you still have the
> problem that the object stored inside the array could be replaced (and
> invalidated) at any time if you run multithreaded.
>

Would it help to have a custom dtype that ensures that all objects in the
array are of this specific extension type? (I don't know if a custom dtype
(done in C, like the quaternion examples) are possible for storing python
objects)


> We would like to type such objects in the future, even then, I am not
> sure how to make things safe against race conditions if elements are
> replaced (and deleted).
>
> This is an interesting use case, since arrays of pointers (or specific
> pyobjects) will always have this type of issue, and I am not sure how
> you would avoid it (a cheap lock on the object itself works probably,
> but even if it is cheap, it is probably fairly expensive?).
>

Currently, we are thinking of doing two loops in the ufunc. First one for
getting all the pointers into a C array, and then manually releasing the
gil for the second loop doing the actual operation on the array of
pointers. See https://github.com/caspervdw/pygeos/issues/7 for example
code. From a quick experiment that seems to give only a small overhead (in
a single threaded case).

That of course still has the same problems as you mentioned (although in
our case, we are, in principle, the holders of the array and know what it
contains, and the individual extension type objects are not mutable), but
then at least it is our own responsibility of making sure that the array
contains the correct objects and is not mutated.

Joris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20190820/9afbe1f6/attachment.html>