[Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way

[David Cournapeau]

On Tue, Sep 22, 2015 at 3:43 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Mon, Sep 21, 2015 at 10:23 PM, Nathaniel Smith <njs at pobox.com> wrote:
>
>> On Mon, Sep 21, 2015 at 7:29 AM, Jaime Fernández del Río
>> <jaime.frio at gmail.com> wrote:
>> > We have the PyArrayObject vs PyArrayObject_fields definition in
>> > ndarraytypes.h that is used to enforce access to the members through
>> inline
>> > functions rather than directly,  which seems to me like the right way
>> to go:
>> > don't leave stones unturned, hide everything and provide PyUFunc_NIN,
>> > PyUFunc_NOUT and friends to handle those too.
>>
>> The PyArrayObject vs PyArrayObject_fields distinction is only enabled
>> if a downstream library explicitly requests it with #define
>> NPY_NO_DEPRECATED_API, though -- the idea is that the changes in this
>> NEP would be enabled unconditionally, even for old code. So the reason
>> nin/nout/nargs are left exposed in this proposal is that there's some
>> existing code out there that would break (until updated) if we hid
>> them, and not much benefit to breaking it.
>>
>> If we're fine with breaking that code then we could just hide them
>> unconditionally too. The only code I found in the wild that would be
>> affected is the "rational" user-defined dtype, which would be
>> trivially fixable since the only thing it does with ufunc->nargs is a
>> quick consistency check:
>>
>>
>> https://github.com/numpy/numpy-dtypes/blob/c0175a6b1c5aa89b4520b29487f06d0e200e2a03/npytypes/rational/rational.c#L1140-L1151
>>
>> Also it's not 100% clear right now whether we even want to keep
>> supporting the old user-defined dtype API that this particular code is
>> based around. But if this code uses ufunc->nargs then perhaps other
>> code does too? I'm open to opinions -- I doubt it matters that much
>> either way. I just want to make sure that we can hide the other stuff
>> :-).
>>
>> When it comes to evolving these APIs in general: one unfortunate thing
>> about the PyArrayObject changes in 1.7 is that because they were
>> implemented using *inline* functions (/macros) they haven't affected
>>
>
> One thing we might consider along the way is separating numpy.multiarray
> and friends into an actual library plus a module. That way the new numpy
> api would be exposed in the library rather than by importing an array of
> pointers from the module.
>

Agreed.

This would help the cythonizing process as well (on which I will try to
write more about in a separate thread later).

David


>
> the a*B*i exposure at all, even in code that has upgraded to the new
>> calling conventions. While user code no longer *names* the internal
>> fields directly, we still have to implement exactly the same fields
>> and put them in exactly the same place in memory or else break ABI.
>> And the other unfortunate thing is that we don't really have a
>> mechanism for saying "okay, we're dropping support for the old way of
>> doing things in 1.xx" -- in particular the current
>> NPY_NO_DEPRECATED_API mechanism doesn't give us any way to detect and
>> error out if someone tries to use an old version of the APIs, so ABI
>> breaks still mean segfaults. I'm thinking that if/when we figure out
>> how to implement the "sliding window" API/ABI idea that we talked
>> about at SciPy, then that will give us a strategy for cleanly
>> transitioning to a world with a maintainable API+ABI and it becomes
>> worth sitting down and making up a set of setters/getters for the
>> attributes that we want to make public in a maintainable way. But
>> until then our only real options are either hard breaks or nothing, so
>> unless we want to do a hard break there's not much point talking about
>> it.
>>
>> -n
>>
>
> <snip>
>
> Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20150922/5fd06541/attachment.html>