On Fri, May 22, 2020 at 10:14 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
Hi all,

just curious, has anyone reservations about extending the ndarray
struct (and the void scalar one)?

The reason is that, I am starting to dislike the way we handle the
buffer interface.
Due to issues with backward compatibility, we cannot use the "right"
way to free the buffer information. Because of that, the way we solve
it is by storing lists of pointers in a dictionary...

To me this seems a bit complicating, and is annoying since it adds a
dictionary lookup overhead to every single array deletion (and
inserting for every buffer creation). Also, it looks a bit like a
memory leak in some cases (although that probably only annoys me and
only when running valgrind).

It seems that it would be much simpler to tag the buffer-info on to the
array object itself. Which, however, would require extending the array
object by a single pointer [1].

Extending is in theory an ABI break if anyone subclasses ndarray from C
(extending the struct) and does not very carefully anticipate the
possibility.  I am not even sure we support that, but its hard to be

I had no idea if we support that, so I crowdsourced some inputs.

Feedback from Travis: "I would be quite sure there are extensions out there that do this.  Please just break the ABI and change the version number to do that."

Feedback from Pearu: "ndarray itself (PyArrayObject) is a kind-of subclass of PyObject. See https://www.python.org/dev/peps/pep-0253.  Something like the following might work:
typedef struct {
PyArrayObject super;
/* insert extensions here */
} MyPyArrayObject;




[1] The size difference should not matter IMO, and with cythons
memoryviews buffers are not an uncommon feature in any case, for the
void scalar it is a bit bigger, but they are also very rare.
(I thought of using weak references, but the CPython API seems not very
fleshed out, or at least not documented, so not sure about that).

NumPy-Discussion mailing list